Participant Presentations Please Sign Up: • Name • Email (Onyen is fine, or …) • Are You ENRolled? • Tentative Title (???? Is OK) • When: Next Week, Early, Oct., Nov., Late
Dec 18, 2015
Participant PresentationsPlease Sign Upbull Namebull Email (Onyen is fine or hellip)bull Are You ENRolledbull Tentative Title ( Is OK)bull When
Next Week Early Oct Nov Late
PCA to find clustersPCA of Mass Flux Data
Statistical SmoothingIn 1 Dimension 2 Major Settings
bull Density EstimationldquoHistogramsrdquo
bull Nonparametric RegressionldquoScatterplot Smoothingrdquo
Kernel Density EstimationChondrite Databull Sum pieces to estimate densitybull Suggests 3 modes (rock sources)
Scatterplot SmoothingEg Bralower Fossil Data ndash some
smooths
Statistical SmoothingFundamental Question
For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot
Smoothingrdquo
Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo
SiZer BackgroundFun Scale Space Views (Incomes Data)
Surface View
SiZer BackgroundSiZer analysis of British Incomes data
PCA to find clustersPCA of Mass Flux Data
Statistical SmoothingIn 1 Dimension 2 Major Settings
bull Density EstimationldquoHistogramsrdquo
bull Nonparametric RegressionldquoScatterplot Smoothingrdquo
Kernel Density EstimationChondrite Databull Sum pieces to estimate densitybull Suggests 3 modes (rock sources)
Scatterplot SmoothingEg Bralower Fossil Data ndash some
smooths
Statistical SmoothingFundamental Question
For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot
Smoothingrdquo
Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo
SiZer BackgroundFun Scale Space Views (Incomes Data)
Surface View
SiZer BackgroundSiZer analysis of British Incomes data
Statistical SmoothingIn 1 Dimension 2 Major Settings
bull Density EstimationldquoHistogramsrdquo
bull Nonparametric RegressionldquoScatterplot Smoothingrdquo
Kernel Density EstimationChondrite Databull Sum pieces to estimate densitybull Suggests 3 modes (rock sources)
Scatterplot SmoothingEg Bralower Fossil Data ndash some
smooths
Statistical SmoothingFundamental Question
For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot
Smoothingrdquo
Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo
SiZer BackgroundFun Scale Space Views (Incomes Data)
Surface View
SiZer BackgroundSiZer analysis of British Incomes data
Kernel Density EstimationChondrite Databull Sum pieces to estimate densitybull Suggests 3 modes (rock sources)
Scatterplot SmoothingEg Bralower Fossil Data ndash some
smooths
Statistical SmoothingFundamental Question
For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot
Smoothingrdquo
Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo
SiZer BackgroundFun Scale Space Views (Incomes Data)
Surface View
SiZer BackgroundSiZer analysis of British Incomes data
Scatterplot SmoothingEg Bralower Fossil Data ndash some
smooths
Statistical SmoothingFundamental Question
For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot
Smoothingrdquo
Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo
SiZer BackgroundFun Scale Space Views (Incomes Data)
Surface View
SiZer BackgroundSiZer analysis of British Incomes data
Statistical SmoothingFundamental Question
For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot
Smoothingrdquo
Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo
SiZer BackgroundFun Scale Space Views (Incomes Data)
Surface View
SiZer BackgroundSiZer analysis of British Incomes data
SiZer BackgroundFun Scale Space Views (Incomes Data)
Surface View
SiZer BackgroundSiZer analysis of British Incomes data
SiZer BackgroundSiZer analysis of British Incomes data
SiZer BackgroundSiZer analysis of British Incomes data bull Oversmoothed Only one mode bull Medium smoothed Two modes
statistically significant
Confirmed by Schmitz amp Marron (1992)bull Undersmoothed many noise
wiggles not significant
Again all are correct
just different scales
SiZer BackgroundScale Space and Kernel Choice
ie Shape of Window
Compelling Answer Gaussian
Only ldquoVariation Diminishingrdquo Kernel Shape
I e Modes decreases with bandwidth h
Lindebergh (1994)
Chaudhuri amp Marron (2000)
SiZer BackgroundRecall
Hidalgo
Stamps
Data
SiZer BackgroundScale Space and Kernel Choice
ie Shape of Window
Compelling Answer Gaussian
Only ldquoVariation Diminishingrdquo Kernel Shape
I e Modes decreases with bandwidth h
Lindebergh (1994)
Chaudhuri amp Marron (2000)
SiZer BackgroundRecall
Hidalgo
Stamps
Data
SiZer BackgroundRecall
Hidalgo
Stamps
Data
SiZer Overview
Would you like to try smoothing amp
SiZer
bull Marron Software Website as Before
bull In ldquoSmoothingrdquo Directory
ndash kdeSMm
ndash nprSMm
ndash sizerSMm
bull Recall ldquogtgt help sizerSMrdquo for
usage
PCA to find clustersReturn to PCA of Mass Flux Data
PCA to find clustersSiZer analysis of Mass Flux PC1
PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot
PCA to find clustersSiZer analysis of Mass Flux PC1
Also in Curvature
PCA to find clustersSiZer analysis of Mass Flux PC1
And in Other Comprsquos
PCA to find clustersSiZer analysis of Mass Flux PC1
Conclusion
bull Found 3 significant clusters
bull Correspond to 3 known ldquocloud typesrdquo
bull Worth deeper investigation
Recall Yeast Cell Cycle Data
bull ldquoGene Expressionrdquo ndash Micro-array data
bull Data (after major preprocessing) Expression ldquolevelrdquo of
bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~
10s)bull Interesting statistical issue
High Dimension Low Sample Size data
(HDLSS)
Yeast Cell Cycle Data FDA View
Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles
Yeast Cell Cycle Data FDA View
Periodic genes
Naiumlve
approach
Simple PCA
Yeast Cell Cycle Data FDA Viewbull Central question which genes are
ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of
ldquovariationrdquobull PCA finds ldquodirections of maximal
variationrdquobull Often but not always same as
ldquointeresting directionsrdquobull Here need better approach to study
periodicities
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
PCA to find clustersReturn to PCA of Mass Flux Data
PCA to find clustersSiZer analysis of Mass Flux PC1
PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot
PCA to find clustersSiZer analysis of Mass Flux PC1
Also in Curvature
PCA to find clustersSiZer analysis of Mass Flux PC1
And in Other Comprsquos
PCA to find clustersSiZer analysis of Mass Flux PC1
Conclusion
bull Found 3 significant clusters
bull Correspond to 3 known ldquocloud typesrdquo
bull Worth deeper investigation
Recall Yeast Cell Cycle Data
bull ldquoGene Expressionrdquo ndash Micro-array data
bull Data (after major preprocessing) Expression ldquolevelrdquo of
bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~
10s)bull Interesting statistical issue
High Dimension Low Sample Size data
(HDLSS)
Yeast Cell Cycle Data FDA View
Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles
Yeast Cell Cycle Data FDA View
Periodic genes
Naiumlve
approach
Simple PCA
Yeast Cell Cycle Data FDA Viewbull Central question which genes are
ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of
ldquovariationrdquobull PCA finds ldquodirections of maximal
variationrdquobull Often but not always same as
ldquointeresting directionsrdquobull Here need better approach to study
periodicities
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
PCA to find clustersSiZer analysis of Mass Flux PC1
PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot
PCA to find clustersSiZer analysis of Mass Flux PC1
Also in Curvature
PCA to find clustersSiZer analysis of Mass Flux PC1
And in Other Comprsquos
PCA to find clustersSiZer analysis of Mass Flux PC1
Conclusion
bull Found 3 significant clusters
bull Correspond to 3 known ldquocloud typesrdquo
bull Worth deeper investigation
Recall Yeast Cell Cycle Data
bull ldquoGene Expressionrdquo ndash Micro-array data
bull Data (after major preprocessing) Expression ldquolevelrdquo of
bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~
10s)bull Interesting statistical issue
High Dimension Low Sample Size data
(HDLSS)
Yeast Cell Cycle Data FDA View
Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles
Yeast Cell Cycle Data FDA View
Periodic genes
Naiumlve
approach
Simple PCA
Yeast Cell Cycle Data FDA Viewbull Central question which genes are
ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of
ldquovariationrdquobull PCA finds ldquodirections of maximal
variationrdquobull Often but not always same as
ldquointeresting directionsrdquobull Here need better approach to study
periodicities
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot
PCA to find clustersSiZer analysis of Mass Flux PC1
Also in Curvature
PCA to find clustersSiZer analysis of Mass Flux PC1
And in Other Comprsquos
PCA to find clustersSiZer analysis of Mass Flux PC1
Conclusion
bull Found 3 significant clusters
bull Correspond to 3 known ldquocloud typesrdquo
bull Worth deeper investigation
Recall Yeast Cell Cycle Data
bull ldquoGene Expressionrdquo ndash Micro-array data
bull Data (after major preprocessing) Expression ldquolevelrdquo of
bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~
10s)bull Interesting statistical issue
High Dimension Low Sample Size data
(HDLSS)
Yeast Cell Cycle Data FDA View
Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles
Yeast Cell Cycle Data FDA View
Periodic genes
Naiumlve
approach
Simple PCA
Yeast Cell Cycle Data FDA Viewbull Central question which genes are
ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of
ldquovariationrdquobull PCA finds ldquodirections of maximal
variationrdquobull Often but not always same as
ldquointeresting directionsrdquobull Here need better approach to study
periodicities
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
PCA to find clustersSiZer analysis of Mass Flux PC1
Also in Curvature
PCA to find clustersSiZer analysis of Mass Flux PC1
And in Other Comprsquos
PCA to find clustersSiZer analysis of Mass Flux PC1
Conclusion
bull Found 3 significant clusters
bull Correspond to 3 known ldquocloud typesrdquo
bull Worth deeper investigation
Recall Yeast Cell Cycle Data
bull ldquoGene Expressionrdquo ndash Micro-array data
bull Data (after major preprocessing) Expression ldquolevelrdquo of
bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~
10s)bull Interesting statistical issue
High Dimension Low Sample Size data
(HDLSS)
Yeast Cell Cycle Data FDA View
Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles
Yeast Cell Cycle Data FDA View
Periodic genes
Naiumlve
approach
Simple PCA
Yeast Cell Cycle Data FDA Viewbull Central question which genes are
ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of
ldquovariationrdquobull PCA finds ldquodirections of maximal
variationrdquobull Often but not always same as
ldquointeresting directionsrdquobull Here need better approach to study
periodicities
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
PCA to find clustersSiZer analysis of Mass Flux PC1
And in Other Comprsquos
PCA to find clustersSiZer analysis of Mass Flux PC1
Conclusion
bull Found 3 significant clusters
bull Correspond to 3 known ldquocloud typesrdquo
bull Worth deeper investigation
Recall Yeast Cell Cycle Data
bull ldquoGene Expressionrdquo ndash Micro-array data
bull Data (after major preprocessing) Expression ldquolevelrdquo of
bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~
10s)bull Interesting statistical issue
High Dimension Low Sample Size data
(HDLSS)
Yeast Cell Cycle Data FDA View
Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles
Yeast Cell Cycle Data FDA View
Periodic genes
Naiumlve
approach
Simple PCA
Yeast Cell Cycle Data FDA Viewbull Central question which genes are
ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of
ldquovariationrdquobull PCA finds ldquodirections of maximal
variationrdquobull Often but not always same as
ldquointeresting directionsrdquobull Here need better approach to study
periodicities
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
PCA to find clustersSiZer analysis of Mass Flux PC1
Conclusion
bull Found 3 significant clusters
bull Correspond to 3 known ldquocloud typesrdquo
bull Worth deeper investigation
Recall Yeast Cell Cycle Data
bull ldquoGene Expressionrdquo ndash Micro-array data
bull Data (after major preprocessing) Expression ldquolevelrdquo of
bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~
10s)bull Interesting statistical issue
High Dimension Low Sample Size data
(HDLSS)
Yeast Cell Cycle Data FDA View
Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles
Yeast Cell Cycle Data FDA View
Periodic genes
Naiumlve
approach
Simple PCA
Yeast Cell Cycle Data FDA Viewbull Central question which genes are
ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of
ldquovariationrdquobull PCA finds ldquodirections of maximal
variationrdquobull Often but not always same as
ldquointeresting directionsrdquobull Here need better approach to study
periodicities
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Recall Yeast Cell Cycle Data
bull ldquoGene Expressionrdquo ndash Micro-array data
bull Data (after major preprocessing) Expression ldquolevelrdquo of
bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~
10s)bull Interesting statistical issue
High Dimension Low Sample Size data
(HDLSS)
Yeast Cell Cycle Data FDA View
Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles
Yeast Cell Cycle Data FDA View
Periodic genes
Naiumlve
approach
Simple PCA
Yeast Cell Cycle Data FDA Viewbull Central question which genes are
ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of
ldquovariationrdquobull PCA finds ldquodirections of maximal
variationrdquobull Often but not always same as
ldquointeresting directionsrdquobull Here need better approach to study
periodicities
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Yeast Cell Cycle Data FDA View
Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles
Yeast Cell Cycle Data FDA View
Periodic genes
Naiumlve
approach
Simple PCA
Yeast Cell Cycle Data FDA Viewbull Central question which genes are
ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of
ldquovariationrdquobull PCA finds ldquodirections of maximal
variationrdquobull Often but not always same as
ldquointeresting directionsrdquobull Here need better approach to study
periodicities
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Yeast Cell Cycle Data FDA View
Periodic genes
Naiumlve
approach
Simple PCA
Yeast Cell Cycle Data FDA Viewbull Central question which genes are
ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of
ldquovariationrdquobull PCA finds ldquodirections of maximal
variationrdquobull Often but not always same as
ldquointeresting directionsrdquobull Here need better approach to study
periodicities
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Yeast Cell Cycle Data FDA Viewbull Central question which genes are
ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of
ldquovariationrdquobull PCA finds ldquodirections of maximal
variationrdquobull Often but not always same as
ldquointeresting directionsrdquobull Here need better approach to study
periodicities
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Yeast Cell Cycles Freq 2 Proj
PCA on
Freq 2
Periodic
Component
Of Data
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase
Approach from Zhao Marron amp Wells (2004)
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Frequency 2 Analysis
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Frequency 2 Analysisbull Project data onto 2-dim space of sin and
cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase
classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but
notable differencesbull Now try to improve ldquophase classificationrdquo
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Yeast Cell CycleRevisit ldquophase classificationrdquo
approachbull Use outer 200 genes
(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis
(finds significant bumps etc in histogram)
bull Carefully redrew boundariesbull Check by studying kde angles
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
SiZer Study of Distrsquon of Angles
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Reclassification of Major Genes
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Compare to Previous Classifrsquon
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
New Subpopulation View
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
New Subpopulation View
NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Detailed Look at PCA
Now Study ldquoFolklorerdquo More Carefully
bull BackGround
bull History
bull Underpinnings
(Mathematical amp Computational)
Good Overall Reference Jolliffe (2002)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
PCA Rediscovery ndash Renaming
Statistics Principal Component Analysis (PCA)
Social Sciences Factor Analysis (PCA is a subset)
Probability Electrical EngKarhunen ndash Loeve expansion
Applied MathematicsProper Orthogonal Decomposition (POD)
Geo-Sciences Empirical Orthogonal Functions (EOF)
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
An Interesting Historical Note
The 1st () application of PCA to Functional
Data Analysis
Rao (1958)
1st Paper with ldquoCurves as Data Objectsrdquo
viewpoint
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
Goal Study Interrelationships
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Detailed Look at PCA
Three Important (amp Interesting) Viewpoints
1 Mathematics
2 Numerics
3 Statistics
1st Review Linear Alg and Multivar Prob
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
xa
i
ii xa
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra
Vector Space
bull set of ldquovectorsrdquo
bull and ldquoscalarsrdquo (coefficients)
bull ldquoclosedrdquo under ldquolinear combinationrdquo
( in space)
eg
ldquo dim Euclidrsquon spacerdquo
xa
i
ii xa
d
d
d xx
x
x
x 1
1
d
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin
Note Planes not Through the Origin
are not Subspaces
(Do not Contain )00 x
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors
(all Linear Combos of them =
= Containing Hyperplane
through Origin)
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Basis of Subspace Set of Vectors that
bull Span ie Everything is a Lin Com of them
bull are Linearly Indeprsquot ie Lin Com is Unique
bull eg ldquoUnit Vector Basisrdquo
bull Since
d
1
0
0
0
1
0
0
0
1
1
0
0
0
1
0
0
0
1
212
1
d
d
xxx
x
x
x
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Basis Matrix of subspace of
Given a basis
create matrix of columns
dnvv 1
nddnd
n
n
vv
vv
vvB
1
111
1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Note Right Multiplication Gives
Linear Combination of Column Vectors
n
iii aBva
1
na
a
a 1
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Then linear combo is a matrix multiplicatrsquon
where
Check sizes
n
iii aBva
1
na
a
a 1
)1()(1 nndd
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Aside on Matrix Multiplication (linear transformatrsquon)
For matrices
Define the Matrix Product
(Inner Products of Rows With Columns )
(Composition of Linear Transformations)
Often Useful to Check Sizes
mkk
m
aa
aa
A
1
111
nmm
n
bb
bb
B
1
111
m
iniik
m
iiik
m
inii
m
iii
baba
baba
AB
1
11
11
111
nmmknk
A B
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Matrix Trace
bull For a Square Matrix
bull Define
bull Trace Commutes with Matrix Multiplication
mmm
m
aa
aa
A
1
111
m
iiiaAtr
1)(
BAtrABtr
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
dd dim
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Dimension of Subspace (a Notion of ldquoSizerdquo)
bull Number of Elements in a Basis (Unique)
bull (Use Basis Above)
bull eg dim of a line is 1
bull eg dim of a plane is 2
bull Dimension is ldquoDegrees of Freedomrdquo
(in Statistical Uses eg ANOVA)
dd dim
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Norm of a Vector
bull in d 21
21
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
d 2121
1
2 xxxx td
jj
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Norm of a Vector
bull in
bull Idea length of the vector
bull Note strange properties for high
eg ldquolength of diagonal of unit cuberdquo =
d 2121
1
2 xxxx td
jj
d
d
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
x
x
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Norm of a Vector (cont)
bull Length Normalized Vector
(has Length 1 thus on Surf of Unit Sphere
amp is a Direction Vector)
bull Define Distance as
x
x
yxyxyxyxd t
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
yxyxyx td
jjj
1
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product
bull for Vectors and
bull Related to Norm via
yxyxyx td
jjj
1
x y
21 xxx
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Inner (Dot Scalar) Product (cont)
bull measures ldquoangle between and rdquo as
bull key to Orthogonality ie Perpendiculrsquoty
if and only if
yyxx
yx
yx
yxyxangle
tt
t
11 cos
cos
x y
yx 0 yx
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Orthonormal Basis
bull All Orthogonal to each other
ie for
bull All have Length 1
ie for
nvv 1
1 ii vv
0 ii vv ii
ni 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
(Coefficient is Inner Product Cool Notation)
nvv 1
n
iii vax
1
ii vxa
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
For the Basis Matrix
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
nvvB 1
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Orthonormal Basis (cont)
bull Spectral Representation
where
Check
bull Matrix Notation where ie
is called transform of
(eg Fourier or Wavelet)
nvv 1
n
iii vax
1
ii vxa
iii
n
iii
n
iiii avvavvavx
1
1
aBx Bxa tt xBa t
xa
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Parseval identity for
in subsp genrsquod by o n basis
bull Pythagorean theorem
bull ldquoDecomposition of Energyrdquo
bull ANOVA - sums of squares
bull Transform has same length as
ie ldquorotation in rdquo
x
nvv 1
2
1
22
1
2 aavxx
n
ii
n
ii
a xd
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
Review of Linear Algebra (Cont)x
xV
V
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Projection of a Vector onto a Subspace
bull Idea Member of that is Closest to
(ie ldquoBest Approxrsquonrdquo)
bull Find that Solves
(ldquoLeast Squaresrdquo)
bull For Inner Product (Hilbert) Space
Exists and is Unique
Review of Linear Algebra (Cont)x
xV
V
VxPV vxVv
min
xPV
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Projection of a Vector onto a Subspace (cont)
bull General Solution in for Basis Matrix
bull So Projrsquon Operator is Matrix Multrsquon
(thus projection is another linear operation)
(note same operation underlies least squares)
Review of Linear Algebra (Cont)
d VB
xBBBBxP tVV
tVVV
1
tVV
tVVV BBBBP
1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis
bull Basis Matrix is Orthonormal
bull So =
= Recon(Coeffs of ldquoin Dirrsquonrdquo)
(Recall Right Multrsquon)
nnVtV IBB
10
01
1
111
1
1
nnn
n
ntn
t
vvvv
vvvv
vv
v
v
xBBxP tVVV
x V
nvv 1
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
V
xPxPx VV 222xPxPx VV
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Projection using Orthonormal Basis (cont)
bull For Orthogonal Complement
and
bull Parseval Inequality
V
xPxPx VV 222xPxPx VV
2
1
22
1
22 aavxxxP
n
ii
n
iiV
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
ddU IUU t
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
(Real) Unitary Matrices with
bull Orthonormal Basis Matrix
(So All of Above Applies)
bull Note Transformrsquon is Distance Preserving
bull Lin Trans (Mult by ) is ~ Rotation
bull But also Includes ldquoMirror Imagesrdquo
ddU IUU t
yxdyxyxyxUyUxUdn
i ii 2
1
U
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find
ndX
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
ndX
ndS
)min(1 ndss
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Singular Value Decomposition (SVD)
For a Matrix
Find a Diagonal Matrix
with Entries
called Singular Values
And Unitary (Rotation) Matrices
(recall )
So That
ndX
ndS
)min(1 ndss
ddU nnV
IVVUU tt tUSVX
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
X
vVSUvVSUvX tt
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
X
vVSUvVSUvX tt
is
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Intuition behind Singular Value Decomposition
bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)
bull First rotate
bull Second rescale coordinate axes (by )
bull Third rotate again
bull ie have diagonalized the transformation
X
vVSUvVSUvX tt
is
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Review of Linear Algebra (Cont)
)min(1 dnss
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
(Since do ldquo0-Stretchingrdquo)
Review of Linear Algebra (Cont)
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
)min(1 dnss
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
r
SVD Compact Representation
Useful Labeling
Singular Values in Increasing Order
Note singular values = 0 can be omitted
Let = of positive singular values
Then
Where are truncations of
trnrrrd VSUX
VSU
)min(1 dnss
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
SVD Full Representation
=
Graphics Display Assumes
ndX ddU ndS nn
tV
nd
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
SVD Full Representation
=
Full Rank Basis Matrix
All 0s in Bottom
ndX ddU ndS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
These Columns Get 0ed Out
ndX ddU nnS nn
tV
nnd 0
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
SVD Reduced Representation
=ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
SVD Reduced Representation
=
Also Some of These May be 0
ndX ndU nnS nn
tV
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
SVD Compact Representation
=
These Get 0ed Out
ndX rdU
rrS nrtV
0
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
SVD Compact Representation
= ndX rdU
rrS nrtV
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find
ddX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
Called Eigenvalues
Convenient Ordering
ddX
d
D
0
01
n 1
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
ddX
d
D
0
01
ddB
ddtt IBBBB
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition
For a (Symmetric) Square Matrix
Find a Diagonal Matrix
And an Orthonormal Matrix
(ie )
So that ie
ddX
d
D
0
01
ddB
ddtt IBBBB
DBBX tBDBX
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )
VU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i
Review of Linear Algebra (Cont)
Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition
(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally
Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with
VU
X
BVU
1i