Top Banner
Participant Presentations Please Sign Up: • Name • Email (Onyen is fine, or …) • Are You ENRolled? • Tentative Title (???? Is OK) • When: Next Week, Early, Oct., Nov., Late
126

Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Dec 18, 2015

Download

Documents

Jemimah George

pc1 slide

simple pca slide

signift slide

curvature slide

usage slide

late slide

smoothing sizer

clusters pca of mass

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Participant PresentationsPlease Sign Upbull Namebull Email (Onyen is fine or hellip)bull Are You ENRolledbull Tentative Title ( Is OK)bull When

Next Week Early Oct Nov Late

PCA to find clustersPCA of Mass Flux Data

Statistical SmoothingIn 1 Dimension 2 Major Settings

bull Density EstimationldquoHistogramsrdquo

bull Nonparametric RegressionldquoScatterplot Smoothingrdquo

Kernel Density EstimationChondrite Databull Sum pieces to estimate densitybull Suggests 3 modes (rock sources)

Scatterplot SmoothingEg Bralower Fossil Data ndash some

smooths

Statistical SmoothingFundamental Question

For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot

Smoothingrdquo

Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo

SiZer BackgroundFun Scale Space Views (Incomes Data)

Surface View

SiZer BackgroundSiZer analysis of British Incomes data

>

SiZer BackgroundSiZer analysis of British Incomes data bull Oversmoothed Only one mode bull Medium smoothed Two modes

statistically significant

Confirmed by Schmitz amp Marron (1992)bull Undersmoothed many noise

wiggles not significant

Again all are correct

just different scales

SiZer BackgroundScale Space and Kernel Choice

ie Shape of Window

Compelling Answer Gaussian

Only ldquoVariation Diminishingrdquo Kernel Shape

I e Modes decreases with bandwidth h

Lindebergh (1994)

Chaudhuri amp Marron (2000)

SiZer BackgroundRecall

Hidalgo

Stamps

Data

>

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 2: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA to find clustersPCA of Mass Flux Data

Statistical SmoothingIn 1 Dimension 2 Major Settings

bull Density EstimationldquoHistogramsrdquo

bull Nonparametric RegressionldquoScatterplot Smoothingrdquo

Kernel Density EstimationChondrite Databull Sum pieces to estimate densitybull Suggests 3 modes (rock sources)

Scatterplot SmoothingEg Bralower Fossil Data ndash some

smooths

Statistical SmoothingFundamental Question

For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot

Smoothingrdquo

Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo

SiZer BackgroundFun Scale Space Views (Incomes Data)

Surface View

SiZer BackgroundSiZer analysis of British Incomes data

>

SiZer BackgroundSiZer analysis of British Incomes data bull Oversmoothed Only one mode bull Medium smoothed Two modes

statistically significant

Confirmed by Schmitz amp Marron (1992)bull Undersmoothed many noise

wiggles not significant

Again all are correct

just different scales

SiZer BackgroundScale Space and Kernel Choice

ie Shape of Window

Compelling Answer Gaussian

Only ldquoVariation Diminishingrdquo Kernel Shape

I e Modes decreases with bandwidth h

Lindebergh (1994)

Chaudhuri amp Marron (2000)

SiZer BackgroundRecall

Hidalgo

Stamps

Data

>

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 3: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Statistical SmoothingIn 1 Dimension 2 Major Settings

bull Density EstimationldquoHistogramsrdquo

bull Nonparametric RegressionldquoScatterplot Smoothingrdquo

Kernel Density EstimationChondrite Databull Sum pieces to estimate densitybull Suggests 3 modes (rock sources)

Scatterplot SmoothingEg Bralower Fossil Data ndash some

smooths

Statistical SmoothingFundamental Question

For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot

Smoothingrdquo

Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo

SiZer BackgroundFun Scale Space Views (Incomes Data)

Surface View

SiZer BackgroundSiZer analysis of British Incomes data

>

SiZer BackgroundSiZer analysis of British Incomes data bull Oversmoothed Only one mode bull Medium smoothed Two modes

statistically significant

Confirmed by Schmitz amp Marron (1992)bull Undersmoothed many noise

wiggles not significant

Again all are correct

just different scales

SiZer BackgroundScale Space and Kernel Choice

ie Shape of Window

Compelling Answer Gaussian

Only ldquoVariation Diminishingrdquo Kernel Shape

I e Modes decreases with bandwidth h

Lindebergh (1994)

Chaudhuri amp Marron (2000)

SiZer BackgroundRecall

Hidalgo

Stamps

Data

>

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 4: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Kernel Density EstimationChondrite Databull Sum pieces to estimate densitybull Suggests 3 modes (rock sources)

Scatterplot SmoothingEg Bralower Fossil Data ndash some

smooths

Statistical SmoothingFundamental Question

For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot

Smoothingrdquo

Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo

SiZer BackgroundFun Scale Space Views (Incomes Data)

Surface View

SiZer BackgroundSiZer analysis of British Incomes data

>

SiZer BackgroundSiZer analysis of British Incomes data bull Oversmoothed Only one mode bull Medium smoothed Two modes

statistically significant

Confirmed by Schmitz amp Marron (1992)bull Undersmoothed many noise

wiggles not significant

Again all are correct

just different scales

SiZer BackgroundScale Space and Kernel Choice

ie Shape of Window

Compelling Answer Gaussian

Only ldquoVariation Diminishingrdquo Kernel Shape

I e Modes decreases with bandwidth h

Lindebergh (1994)

Chaudhuri amp Marron (2000)

SiZer BackgroundRecall

Hidalgo

Stamps

Data

>

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 5: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Scatterplot SmoothingEg Bralower Fossil Data ndash some

smooths

Statistical SmoothingFundamental Question

For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot

Smoothingrdquo

Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo

SiZer BackgroundFun Scale Space Views (Incomes Data)

Surface View

SiZer BackgroundSiZer analysis of British Incomes data

>

SiZer BackgroundSiZer analysis of British Incomes data bull Oversmoothed Only one mode bull Medium smoothed Two modes

statistically significant

Confirmed by Schmitz amp Marron (1992)bull Undersmoothed many noise

wiggles not significant

Again all are correct

just different scales

SiZer BackgroundScale Space and Kernel Choice

ie Shape of Window

Compelling Answer Gaussian

Only ldquoVariation Diminishingrdquo Kernel Shape

I e Modes decreases with bandwidth h

Lindebergh (1994)

Chaudhuri amp Marron (2000)

SiZer BackgroundRecall

Hidalgo

Stamps

Data

>

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 6: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Statistical SmoothingFundamental Question

For both ofbull Density Estimation ldquoHistogramsrdquobull Regression ldquoScatterplot

Smoothingrdquo

Which bumps are ldquoreally thererdquovs ldquoartifacts of sampling noiserdquo

SiZer BackgroundFun Scale Space Views (Incomes Data)

Surface View

SiZer BackgroundSiZer analysis of British Incomes data

>

SiZer BackgroundSiZer analysis of British Incomes data bull Oversmoothed Only one mode bull Medium smoothed Two modes

statistically significant

Confirmed by Schmitz amp Marron (1992)bull Undersmoothed many noise

wiggles not significant

Again all are correct

just different scales

SiZer BackgroundScale Space and Kernel Choice

ie Shape of Window

Compelling Answer Gaussian

Only ldquoVariation Diminishingrdquo Kernel Shape

I e Modes decreases with bandwidth h

Lindebergh (1994)

Chaudhuri amp Marron (2000)

SiZer BackgroundRecall

Hidalgo

Stamps

Data

>

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 7: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

SiZer BackgroundFun Scale Space Views (Incomes Data)

Surface View

SiZer BackgroundSiZer analysis of British Incomes data

>

SiZer BackgroundSiZer analysis of British Incomes data bull Oversmoothed Only one mode bull Medium smoothed Two modes

statistically significant

Confirmed by Schmitz amp Marron (1992)bull Undersmoothed many noise

wiggles not significant

Again all are correct

just different scales

SiZer BackgroundScale Space and Kernel Choice

ie Shape of Window

Compelling Answer Gaussian

Only ldquoVariation Diminishingrdquo Kernel Shape

I e Modes decreases with bandwidth h

Lindebergh (1994)

Chaudhuri amp Marron (2000)

SiZer BackgroundRecall

Hidalgo

Stamps

Data

>

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 8: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

SiZer BackgroundSiZer analysis of British Incomes data

>

SiZer BackgroundSiZer analysis of British Incomes data bull Oversmoothed Only one mode bull Medium smoothed Two modes

statistically significant

Confirmed by Schmitz amp Marron (1992)bull Undersmoothed many noise

wiggles not significant

Again all are correct

just different scales

SiZer BackgroundScale Space and Kernel Choice

ie Shape of Window

Compelling Answer Gaussian

Only ldquoVariation Diminishingrdquo Kernel Shape

I e Modes decreases with bandwidth h

Lindebergh (1994)

Chaudhuri amp Marron (2000)

SiZer BackgroundRecall

Hidalgo

Stamps

Data

>

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 9: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

SiZer BackgroundSiZer analysis of British Incomes data bull Oversmoothed Only one mode bull Medium smoothed Two modes

statistically significant

Confirmed by Schmitz amp Marron (1992)bull Undersmoothed many noise

wiggles not significant

Again all are correct

just different scales

SiZer BackgroundScale Space and Kernel Choice

ie Shape of Window

Compelling Answer Gaussian

Only ldquoVariation Diminishingrdquo Kernel Shape

I e Modes decreases with bandwidth h

Lindebergh (1994)

Chaudhuri amp Marron (2000)

SiZer BackgroundRecall

Hidalgo

Stamps

Data

>

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 10: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

SiZer BackgroundScale Space and Kernel Choice

ie Shape of Window

Compelling Answer Gaussian

Only ldquoVariation Diminishingrdquo Kernel Shape

I e Modes decreases with bandwidth h

Lindebergh (1994)

Chaudhuri amp Marron (2000)

SiZer BackgroundRecall

Hidalgo

Stamps

Data

>

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 11: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

SiZer BackgroundRecall

Hidalgo

Stamps

Data

>

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 12: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

SiZer Overview

Would you like to try smoothing amp

SiZer

bull Marron Software Website as Before

bull In ldquoSmoothingrdquo Directory

ndash kdeSMm

ndash nprSMm

ndash sizerSMm

bull Recall ldquogtgt help sizerSMrdquo for

usage

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 13: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA to find clustersReturn to PCA of Mass Flux Data

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 14: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA to find clustersSiZer analysis of Mass Flux PC1

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 15: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA to find clustersSiZer analysis of Mass Flux PC1All 3Signifrsquot

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 16: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA to find clustersSiZer analysis of Mass Flux PC1

Also in Curvature

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 17: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA to find clustersSiZer analysis of Mass Flux PC1

And in Other Comprsquos

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 18: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA to find clustersSiZer analysis of Mass Flux PC1

Conclusion

bull Found 3 significant clusters

bull Correspond to 3 known ldquocloud typesrdquo

bull Worth deeper investigation

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 19: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Recall Yeast Cell Cycle Data

bull ldquoGene Expressionrdquo ndash Micro-array data

bull Data (after major preprocessing) Expression ldquolevelrdquo of

bull thousands of genes (d ~ 1000s)bull but only dozens of ldquocasesrdquo (n ~

10s)bull Interesting statistical issue

High Dimension Low Sample Size data

(HDLSS)

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 20: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Yeast Cell Cycle Data FDA View

Central questionWhich genes are ldquoperiodicrdquo over 2 cell cycles

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 21: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Yeast Cell Cycle Data FDA View

Periodic genes

Naiumlve

approach

Simple PCA

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 22: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Yeast Cell Cycle Data FDA Viewbull Central question which genes are

ldquoperiodicrdquo over 2 cell cyclesbull Naiumlve approach Simple PCAbull No apparent (2 cycle) periodic structurebull Eigenvalues suggest large amount of

ldquovariationrdquobull PCA finds ldquodirections of maximal

variationrdquobull Often but not always same as

ldquointeresting directionsrdquobull Here need better approach to study

periodicities

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 23: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Yeast Cell Cycles Freq 2 Proj

PCA on

Freq 2

Periodic

Component

Of Data

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 24: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phase

Approach from Zhao Marron amp Wells (2004)

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 25: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Frequency 2 Analysis

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 26: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Frequency 2 Analysisbull Project data onto 2-dim space of sin and

cos (freq 2)bull Useful view scatterplotbull Angle (in polar coordinates) shows phasebull Colors Spellmanrsquos cell cycle phase

classificationbull Black was labeled ldquonot periodicrdquobull Within class phases approxrsquoly same but

notable differencesbull Now try to improve ldquophase classificationrdquo

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 27: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Yeast Cell CycleRevisit ldquophase classificationrdquo

approachbull Use outer 200 genes

(other numbers tried less resolution)bull Study distribution of anglesbull Use SiZer analysis

(finds significant bumps etc in histogram)

bull Carefully redrew boundariesbull Check by studying kde angles

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 28: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

SiZer Study of Distrsquon of Angles

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 29: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Reclassification of Major Genes

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 30: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Compare to Previous Classifrsquon

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 31: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

New Subpopulation View

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 32: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

New Subpopulation View

NoteSubdensitiesHave SameBandwidth ampProportionalAreas(so Σ = 1)

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 33: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Detailed Look at PCA

Now Study ldquoFolklorerdquo More Carefully

bull BackGround

bull History

bull Underpinnings

(Mathematical amp Computational)

Good Overall Reference Jolliffe (2002)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 34: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 35: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 36: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 37: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 38: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

PCA Rediscovery ndash Renaming

Statistics Principal Component Analysis (PCA)

Social Sciences Factor Analysis (PCA is a subset)

Probability Electrical EngKarhunen ndash Loeve expansion

Applied MathematicsProper Orthogonal Decomposition (POD)

Geo-Sciences Empirical Orthogonal Functions (EOF)

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 39: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 40: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

An Interesting Historical Note

The 1st () application of PCA to Functional

Data Analysis

Rao (1958)

1st Paper with ldquoCurves as Data Objectsrdquo

viewpoint

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 41: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

Goal Study Interrelationships

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 42: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Detailed Look at PCA

Three Important (amp Interesting) Viewpoints

1 Mathematics

2 Numerics

3 Statistics

1st Review Linear Alg and Multivar Prob

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 43: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 44: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

xa

i

ii xa

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 45: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra

Vector Space

bull set of ldquovectorsrdquo

bull and ldquoscalarsrdquo (coefficients)

bull ldquoclosedrdquo under ldquolinear combinationrdquo

( in space)

eg

ldquo dim Euclidrsquon spacerdquo

xa

i

ii xa

d

d

d xx

x

x

x 1

1

d

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 46: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combination

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 47: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 48: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Origin

Note Planes not Through the Origin

are not Subspaces

(Do not Contain )00 x

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 49: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Subspacebull Subset that is Again a Vector Spacebull ie Closed under Linear Combinationbull eg Lines through the Originbull eg Planes through the Originbull eg Subsp ldquoGenerated Byrdquo a Set of Vectors

(all Linear Combos of them =

= Containing Hyperplane

through Origin)

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 50: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 51: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Basis of Subspace Set of Vectors that

bull Span ie Everything is a Lin Com of them

bull are Linearly Indeprsquot ie Lin Com is Unique

bull eg ldquoUnit Vector Basisrdquo

bull Since

d

1

0

0

0

1

0

0

0

1

1

0

0

0

1

0

0

0

1

212

1

d

d

xxx

x

x

x

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 52: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Basis Matrix of subspace of

Given a basis

create matrix of columns

dnvv 1

nddnd

n

n

vv

vv

vvB

1

111

1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 53: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 54: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Note Right Multiplication Gives

Linear Combination of Column Vectors

n

iii aBva

1

na

a

a 1

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 55: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Then linear combo is a matrix multiplicatrsquon

where

Check sizes

n

iii aBva

1

na

a

a 1

)1()(1 nndd

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 56: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 57: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 58: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 59: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 60: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Aside on Matrix Multiplication (linear transformatrsquon)

For matrices

Define the Matrix Product

(Inner Products of Rows With Columns )

(Composition of Linear Transformations)

Often Useful to Check Sizes

mkk

m

aa

aa

A

1

111

nmm

n

bb

bb

B

1

111

m

iniik

m

iiik

m

inii

m

iii

baba

baba

AB

1

11

11

111

nmmknk

A B

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 61: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 62: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Matrix Trace

bull For a Square Matrix

bull Define

bull Trace Commutes with Matrix Multiplication

mmm

m

aa

aa

A

1

111

m

iiiaAtr

1)(

BAtrABtr

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 63: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 64: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

dd dim

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 65: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Dimension of Subspace (a Notion of ldquoSizerdquo)

bull Number of Elements in a Basis (Unique)

bull (Use Basis Above)

bull eg dim of a line is 1

bull eg dim of a plane is 2

bull Dimension is ldquoDegrees of Freedomrdquo

(in Statistical Uses eg ANOVA)

dd dim

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 66: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Norm of a Vector

bull in d 21

21

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 67: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

d 2121

1

2 xxxx td

jj

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 68: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Norm of a Vector

bull in

bull Idea length of the vector

bull Note strange properties for high

eg ldquolength of diagonal of unit cuberdquo =

d 2121

1

2 xxxx td

jj

d

d

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 69: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

x

x

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 70: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Norm of a Vector (cont)

bull Length Normalized Vector

(has Length 1 thus on Surf of Unit Sphere

amp is a Direction Vector)

bull Define Distance as

x

x

yxyxyxyxd t

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 71: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

yxyxyx td

jjj

1

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 72: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product

bull for Vectors and

bull Related to Norm via

yxyxyx td

jjj

1

x y

21 xxx

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 73: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 74: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Inner (Dot Scalar) Product (cont)

bull measures ldquoangle between and rdquo as

bull key to Orthogonality ie Perpendiculrsquoty

if and only if

yyxx

yx

yx

yxyxangle

tt

t

11 cos

cos

x y

yx 0 yx

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 75: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Orthonormal Basis

bull All Orthogonal to each other

ie for

bull All have Length 1

ie for

nvv 1

1 ii vv

0 ii vv ii

ni 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 76: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 77: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

(Coefficient is Inner Product Cool Notation)

nvv 1

n

iii vax

1

ii vxa

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 78: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 79: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

For the Basis Matrix

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

nvvB 1

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 80: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Orthonormal Basis (cont)

bull Spectral Representation

where

Check

bull Matrix Notation where ie

is called transform of

(eg Fourier or Wavelet)

nvv 1

n

iii vax

1

ii vxa

iii

n

iii

n

iiii avvavvavx

1

1

aBx Bxa tt xBa t

xa

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 81: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 82: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 83: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Parseval identity for

in subsp genrsquod by o n basis

bull Pythagorean theorem

bull ldquoDecomposition of Energyrdquo

bull ANOVA - sums of squares

bull Transform has same length as

ie ldquorotation in rdquo

x

nvv 1

2

1

22

1

2 aavxx

n

ii

n

ii

a xd

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 84: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

Review of Linear Algebra (Cont)x

xV

V

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 85: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 86: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Projection of a Vector onto a Subspace

bull Idea Member of that is Closest to

(ie ldquoBest Approxrsquonrdquo)

bull Find that Solves

(ldquoLeast Squaresrdquo)

bull For Inner Product (Hilbert) Space

Exists and is Unique

Review of Linear Algebra (Cont)x

xV

V

VxPV vxVv

min

xPV

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 87: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 88: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 89: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Projection of a Vector onto a Subspace (cont)

bull General Solution in for Basis Matrix

bull So Projrsquon Operator is Matrix Multrsquon

(thus projection is another linear operation)

(note same operation underlies least squares)

Review of Linear Algebra (Cont)

d VB

xBBBBxP tVV

tVVV

1

tVV

tVVV BBBBP

1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 90: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 91: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis

bull Basis Matrix is Orthonormal

bull So =

= Recon(Coeffs of ldquoin Dirrsquonrdquo)

(Recall Right Multrsquon)

nnVtV IBB

10

01

1

111

1

1

nnn

n

ntn

t

vvvv

vvvv

vv

v

v

xBBxP tVVV

x V

nvv 1

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 92: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

V

xPxPx VV 222xPxPx VV

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 93: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Projection using Orthonormal Basis (cont)

bull For Orthogonal Complement

and

bull Parseval Inequality

V

xPxPx VV 222xPxPx VV

2

1

22

1

22 aavxxxP

n

ii

n

iiV

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 94: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

ddU IUU t

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 95: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 96: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

(Real) Unitary Matrices with

bull Orthonormal Basis Matrix

(So All of Above Applies)

bull Note Transformrsquon is Distance Preserving

bull Lin Trans (Mult by ) is ~ Rotation

bull But also Includes ldquoMirror Imagesrdquo

ddU IUU t

yxdyxyxyxUyUxUdn

i ii 2

1

U

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 97: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find

ndX

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 98: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

ndX

ndS

)min(1 ndss

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 99: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 100: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Singular Value Decomposition (SVD)

For a Matrix

Find a Diagonal Matrix

with Entries

called Singular Values

And Unitary (Rotation) Matrices

(recall )

So That

ndX

ndS

)min(1 ndss

ddU nnV

IVVUU tt tUSVX

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 101: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 102: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

X

vVSUvVSUvX tt

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 103: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 104: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

X

vVSUvVSUvX tt

is

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 105: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Intuition behind Singular Value Decomposition

bull For a ldquolinear transfrsquonrdquo (via matrix multirsquon)

bull First rotate

bull Second rescale coordinate axes (by )

bull Third rotate again

bull ie have diagonalized the transformation

X

vVSUvVSUvX tt

is

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 106: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Review of Linear Algebra (Cont)

)min(1 dnss

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 107: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

(Since do ldquo0-Stretchingrdquo)

Review of Linear Algebra (Cont)

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 108: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

)min(1 dnss

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 109: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

r

SVD Compact Representation

Useful Labeling

Singular Values in Increasing Order

Note singular values = 0 can be omitted

Let = of positive singular values

Then

Where are truncations of

trnrrrd VSUX

VSU

)min(1 dnss

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 110: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

SVD Full Representation

=

Graphics Display Assumes

ndX ddU ndS nn

tV

nd

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 111: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 112: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

SVD Full Representation

=

Full Rank Basis Matrix

All 0s in Bottom

ndX ddU ndS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 113: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

These Columns Get 0ed Out

ndX ddU nnS nn

tV

nnd 0

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 114: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

SVD Reduced Representation

=ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 115: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

SVD Reduced Representation

=

Also Some of These May be 0

ndX ndU nnS nn

tV

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 116: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 117: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

SVD Compact Representation

=

These Get 0ed Out

ndX rdU

rrS nrtV

0

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 118: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

SVD Compact Representation

= ndX rdU

rrS nrtV

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 119: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find

ddX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 120: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

Called Eigenvalues

Convenient Ordering

ddX

d

D

0

01

n 1

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 121: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

ddX

d

D

0

01

ddB

ddtt IBBBB

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 122: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Eigenvalue Decomposition

For a (Symmetric) Square Matrix

Find a Diagonal Matrix

And an Orthonormal Matrix

(ie )

So that ie

ddX

d

D

0

01

ddB

ddtt IBBBB

DBBX tBDBX

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 123: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 124: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs VU

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 125: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )

VU

1i

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)
Page 126: Participant Presentations Please Sign Up: Name Email (Onyen is fine, or …) Are You ENRolled? Tentative Title (???? Is OK) When: Next Week, Early, Oct.,

Review of Linear Algebra (Cont)

Eigenvalue Decomposition (cont)bull Relation to Singular Value Decomposition

(looks similar)bull Eigenvalue Decomposition ldquoLooks Harderrdquobull Since Needs bull Price is Eigenvalue Decomprsquon is Generally

Complex (uses )bull Except for Square and Symmetricbull Then Eigenvalue Decomp is Real Valuedbull Thus is the Singrsquor Value Decomp with

VU

X

BVU

1i

  • Participant Presentations
  • PCA to find clusters
  • Statistical Smoothing
  • Kernel Density Estimation
  • Scatterplot Smoothing
  • Statistical Smoothing (2)
  • SiZer Background
  • SiZer Background (2)
  • SiZer Background (3)
  • SiZer Background (4)
  • SiZer Background (5)
  • SiZer Overview
  • PCA to find clusters (2)
  • PCA to find clusters (3)
  • PCA to find clusters (4)
  • PCA to find clusters (5)
  • PCA to find clusters (6)
  • PCA to find clusters (7)
  • Recall Yeast Cell Cycle Data
  • Yeast Cell Cycle Data FDA View
  • Yeast Cell Cycle Data FDA View (2)
  • Yeast Cell Cycle Data FDA View (3)
  • Yeast Cell Cycles Freq 2 Proj
  • Frequency 2 Analysis
  • Frequency 2 Analysis (2)
  • Frequency 2 Analysis (3)
  • Yeast Cell Cycle
  • SiZer Study of Distrsquon of Angles
  • Reclassification of Major Genes
  • Compare to Previous Classifrsquon
  • New Subpopulation View
  • New Subpopulation View (2)
  • Detailed Look at PCA
  • PCA Rediscovery ndash Renaming
  • PCA Rediscovery ndash Renaming (2)
  • PCA Rediscovery ndash Renaming (3)
  • PCA Rediscovery ndash Renaming (4)
  • PCA Rediscovery ndash Renaming (5)
  • An Interesting Historical Note
  • An Interesting Historical Note (2)
  • Detailed Look at PCA (2)
  • Detailed Look at PCA (3)
  • Review of Linear Algebra
  • Review of Linear Algebra (2)
  • Review of Linear Algebra (3)
  • Review of Linear Algebra (Cont)
  • Review of Linear Algebra (Cont) (2)
  • Review of Linear Algebra (Cont) (3)
  • Review of Linear Algebra (Cont) (4)
  • Review of Linear Algebra (Cont) (5)
  • Review of Linear Algebra (Cont) (6)
  • Review of Linear Algebra (Cont) (7)
  • Review of Linear Algebra (Cont) (8)
  • Review of Linear Algebra (Cont) (9)
  • Review of Linear Algebra (Cont) (10)
  • Review of Linear Algebra (Cont) (11)
  • Review of Linear Algebra (Cont) (12)
  • Review of Linear Algebra (Cont) (13)
  • Review of Linear Algebra (Cont) (14)
  • Review of Linear Algebra (Cont) (15)
  • Review of Linear Algebra (Cont) (16)
  • Review of Linear Algebra (Cont) (17)
  • Review of Linear Algebra (Cont) (18)
  • Review of Linear Algebra (Cont) (19)
  • Review of Linear Algebra (Cont) (20)
  • Review of Linear Algebra (Cont) (21)
  • Review of Linear Algebra (Cont) (22)
  • Review of Linear Algebra (Cont) (23)
  • Review of Linear Algebra (Cont) (24)
  • Review of Linear Algebra (Cont) (25)
  • Review of Linear Algebra (Cont) (26)
  • Review of Linear Algebra (Cont) (27)
  • Review of Linear Algebra (Cont) (28)
  • Review of Linear Algebra (Cont) (29)
  • Review of Linear Algebra (Cont) (30)
  • Review of Linear Algebra (Cont) (31)
  • Review of Linear Algebra (Cont) (32)
  • Review of Linear Algebra (Cont) (33)
  • Review of Linear Algebra (Cont) (34)
  • Review of Linear Algebra (Cont) (35)
  • Review of Linear Algebra (Cont) (36)
  • Review of Linear Algebra (Cont) (37)
  • Review of Linear Algebra (Cont) (38)
  • Review of Linear Algebra (Cont) (39)
  • Review of Linear Algebra (Cont) (40)
  • Review of Linear Algebra (Cont) (41)
  • Review of Linear Algebra (Cont) (42)
  • Review of Linear Algebra (Cont) (43)
  • Review of Linear Algebra (Cont) (44)
  • Review of Linear Algebra (Cont) (45)
  • Review of Linear Algebra (Cont) (46)
  • Review of Linear Algebra (Cont) (47)
  • Review of Linear Algebra (Cont) (48)
  • Review of Linear Algebra (Cont) (49)
  • Review of Linear Algebra (Cont) (50)
  • Review of Linear Algebra (Cont) (51)
  • Review of Linear Algebra (Cont) (52)
  • Review of Linear Algebra (Cont) (53)
  • Review of Linear Algebra (Cont) (54)
  • Review of Linear Algebra (Cont) (55)
  • Review of Linear Algebra (Cont) (56)
  • Review of Linear Algebra (Cont) (57)
  • Review of Linear Algebra (Cont) (58)
  • Review of Linear Algebra (Cont) (59)
  • Review of Linear Algebra (Cont) (60)
  • Review of Linear Algebra (Cont) (61)
  • Review of Linear Algebra (Cont) (62)
  • Review of Linear Algebra (Cont) (63)
  • Review of Linear Algebra (Cont) (64)
  • Review of Linear Algebra (Cont) (65)
  • Review of Linear Algebra (Cont) (66)
  • Review of Linear Algebra (Cont) (67)
  • Review of Linear Algebra (Cont) (68)
  • Review of Linear Algebra (Cont) (69)
  • Review of Linear Algebra (Cont) (70)
  • Review of Linear Algebra (Cont) (71)
  • Review of Linear Algebra (Cont) (72)
  • Review of Linear Algebra (Cont) (73)
  • Review of Linear Algebra (Cont) (74)
  • Review of Linear Algebra (Cont) (75)
  • Review of Linear Algebra (Cont) (76)
  • Review of Linear Algebra (Cont) (77)
  • Review of Linear Algebra (Cont) (78)
  • Review of Linear Algebra (Cont) (79)
  • Review of Linear Algebra (Cont) (80)
  • Review of Linear Algebra (Cont) (81)

Related Documents
Tentative teachers contract

Tentative teachers contract

Category: Education
Woods Tentative

Woods Tentative

Category: Documents
Final Tentative Agreement

Final Tentative Agreement

Category: Documents
BIRMINGHAM EDUCATION ASSOCIATION TENTATIVE AGREEMENT FOR ...birminghamea.org/docs/BEA-BPS TENTATIVE AGREEMENT 2015-18.pdf · BIRMINGHAM EDUCATION ASSOCIATION TENTATIVE AGREEMENT FOR

BIRMINGHAM EDUCATION ASSOCIATION TENTATIVE AGREEMENT FOR...

Category: Documents
Tentative Product

Tentative Product

Category: Documents
DENISON MINES ZAMBIA LIMITED ENVIRONMENTAL ... - ONYEN®

DENISON MINES ZAMBIA LIMITED ENVIRONMENTAL ... - ONYEN®

Category: Documents
Tentative Final Monograph

Tentative Final Monograph

Category: Documents
2017 – 2018 5M 6 Tentative Schedule – 2018 5M-6 Tentative Schedule 2018 Tentative 2017 Tentative January 24th June 30rd - July 4th 100th Intl. onvention/hicago July 18th abinet

2017 – 2018 5M 6 Tentative Schedule – 2018 5M-6...

Category: Documents
Tentative Programme Schedule

Tentative Programme Schedule

Category: Documents
Tentative verdict form1

Tentative verdict form1

Category: Devices & Hardware
 · 2019-08-19 · Tentative Tentative Tentative Tentative Tentative anel for romotion of Rece tionist to Su ervisor :- EMPID Name & Parentage No. 10 11 12 390 365 438 386 389 421

 · 2019-08-19 · Tentative Tentative Tentative Tentative....

Category: Documents
Appendix D: Geotechnical Investigation, Tentative Tract ... · Appendix D: Geotechnical Investigation, Tentative Tract Number 5928, ... Geotechnical Investigation . Tentative Tract

Appendix D: Geotechnical Investigation, Tentative Tract...

Category: Documents