Examples Done on Orthogonal Projection

8/18/2019 Examples Done on Orthogonal Projection

1/79

1

Projection and Some of Its Applications

Mohammed Nasser

Professor, Dept. of Statistics, RU,BangladeshEmail: [email protected]

1

The use of matrix theory is now widespread .- - - -- areessential in ----------modern treatment of univeriate andmultivariate statistical methods. ----------C. . ao


2/79

!

"#li$ue and "rtho%onal Projection in !

"rtho%onal Projection into a &ine n

Inner product SpaceProjection into a Su#space'ram-Schmidt "rtho%onali(ationProjection and )atricesProjection in Infinite-dimensional SpaceProjection in )ultivariate )ethods

Contents

!


3/79

*

M a t h em

a t i c al

C on

c e p t s


4/79

+

Mathematical Concepts

Covariance

VarianceProjection


5/79

,

1

0 and

1

1

1."#li$ue and "rtho%onalProjection in !

are two independent vectors in!

1 2

1 0{ | } and { | }

1 1V l l R V m m R⇒ ∈ ∈

are two one-dimensional su#spaces in !

! 1 / !1 0 ! 23

! 1 !


6/79

4


51617 1

52617

!


7/79

8

1

0 and

1

1


are two independent vectors in!

=⇔=⇔+=⇒−

2

1

2

11

2

1

2

121

2

1

1 1

0 1

1 1

0 1

1

0

1

1

a

a

x

x

a

a

x

xaa

x

x

−=⇒ 121

2

1

x x x

aa

−+=⇒1

0)(

1

1121

2

1 x x x x

x


8/79

9


:e define;&; ! as → 11

t =1

11

2

1 x

x

x L

:e can easily show that it is a linear mapThis linear map is called projection 5o#li$ue


9/79

=


51617 1

!

5-1617


10/79

12


1

1- and

1

1 are two ortho%onal5>independent7 vectors in !

&et us consider −

+= 11

1

121

2

1

aa x

x

In this case we can find values of ?a@ withoutinverse

[ ] [ ]

[ ]

[ ]2

2

1

1

1

2

1

111 1

1 1

111 11 1

v

v x x

x

a

a x x

T

==

=


11/79

11


:e define;&; ! as → 11

t = 11

1

1

1

1

2

2

1

T

x

x

x L

This linear map is called ortho%onalprojection.

The vector is projected on the space %enerated #yalon% the space %enerated #y 1

1

−1

1


12/79

1!

Projections

=

2

2

2

000

010

001

0

2

2

5262617

5261627

5162627

5!6!6!7

a 51627

# 5!6!7

==0

2a

aaba

c T T


13/79

1*

1."rtho%onal Projection Into a &ine

Definition 1.1 : Orthogonal Projection

The orthogonal projection of v into the line spanned by a non ero s is the!ector.

( ) ( )"" proj = ×s v v s s

×= ×v s

ss s

" = =×

s ss

s s s

#$a%ple 1.1 : Orthogonal projection of the !ector ( 2 & ) T into the line y '2 x.

( )2 11 1

2 & 2

proj = + ÷ ÷

s

2 x x = ÷

s 2 2* x x x= + =s 11"2 = ÷

s

1+2 = ÷ 1*

If S has unit len%th6 then 5v.s7s and its len%th is 5v.s7

( ) ( )"" proj = ×s v v s s


14/79

1+

#$a%ple 1.2 : Orthogonal projection of a general !ector in , & into the y-a$is

2

0

10

÷

= ÷ ÷ e 2

0 0

1 10 0

x x

proj y y z z

÷ ÷ ÷ ÷

= × ÷ ÷ ÷ ÷ ÷ ÷ ÷ ÷ e

0

10

y

÷

= ÷ ÷

0

0 y

÷

= ÷ ÷

#$a%ple 1.& : Project ' Discard orthogonal co%ponents railroad car left on an east- est trac/ itho t its bra/e is p shed by a ind

blo ing to ard the northeast at fifteen %iles per ho r hat speed ill the car reach

1 proj= ev w 11 02 = ÷

1112

= ÷ w

12

speed = =v

1+


15/79

1,

#$a%ple 1.* : 3earest Point

1,

5a6#7 A

C

5c6d7

&et A 5a6#7 and 5c6d7 #etwo vectors. :e have tofind the nearest vector to Aon

That means we have to find the value of B for which5B7 55a6#7- B5c6d77T55a6#7- B5c6d77 is minimum 6 i.e. 6 len%th

of AC is minimum.

D5c6d7

Easy application of derivative shows that

5A. 7F . 3


16/79

14

Exercises 1

1. Consider the function mappin% a plane to itself thattaBes a vector to its

projection into the line y x .5a7Produce a matrix that descri#es the functionGs action.5#7Show also that this map can #e o#tained #y first

rotatin% everythin% in the plane HF+ radians clocBwise6then projectin% into the x -axis6 and then rotatin% HF+radians counterclocBwise.

!. Show that

14

0))().(( =− v projvv proj s s


17/79

18

2.1Definition

n inner product on a real spaces V is a f nction that associatesa n %ber4 denoted u4v 4 ith each pair of !ectors u and v of V . This f nction has to satisfy the follo ing conditions for

!ectors u4v4 and w4 and scalar c.

1. u4v ' v4u (sy%%etry a$io%)2. u + v 4w ' u4w 5 v4w (additi!e a$io%)&. cu4v ' c u4v (ho%ogeneity a$io%)*. u4u ≥ 04 and u4u ' 0 if and only if u ' 0

(position definite a$io%)

2.Inner product pace


18/79

19

vector space on which an inner product is defined iscalled an inner product space .Any function on a vectorspace that satisfies the axioms of an inner product defines

an inner product on the space. .

There can #e many inner products on a %iven vectorspace

2.Inner product pace


19/79

1=

Example !.16et u ' ( x14 x2)4v ' ( y14 y2)4 and w ' ( z 14 z 2) be arbitrary !ectors in

R 2

. Pro!e that u4v 4 defined as follo s4 is an inner prod cton R 2.

u4v ' x1 y1 5 * x2 y2Deter%ine the inner prod ct of the !ectors ( −24 )4 (&4 1) nder

this inner prod ct.olution

$io% 1: u4v ' x1 y1 5 * x2 y2 ' y1 x1 5 * y2 x2 ' v4u

$io% 2: u + v 4w ' ( x14 x2) 5 ( y14 y2) 4 ( z 14 z 2)

' ( x1 5 y14 x2 5 y2)4 ( z 14 z 2)' ( x1 5 y1) z 1 5 *( x2 5 y2) z 2' x1 z 1 5 * x2 z 2 5 y1 z 1 5 * y2 z 2

' ( x14 x2)4 ( z 14 z 2) 5 ( y14 y2)4 ( z 14 z 2)'


20/79

!2

$io% &: cu4v ' c( x14 x2)4 ( y14 y2)

' (cx14cx2)4 ( y14 y2) ' cx1 y1 5 * cx2 y2 ' c( x1 y1 5 * x2 y2)

' c u4v $io% *: u4u ' ( x14 x2)4 ( x14 x2) ' 0*

22

21 ≥+ x x

7 rther4 if and only if x1 ' 0 and x2 ' 0. That is u '0. Th s u4u ≥ 04 and u4u ' 0 if and only if u ' 0.The fo r inner prod ct a$io%s are satisfied4

u4v ' x1 y1 5 * x2 y2 is an inner prod ct on R 2.

0*22

21 =+ x x

The inner prod ct of the !ectors ( −24 )4 (&4 1) is(−24 )4 (&4 1) ' ( −2 × &) 5 *( × 1) ' 1*


21/79

!1

Example !.!8onsider the !ector space M 22 of 2 × 2 %atrices. 6et u and v defined as follo s be arbitrary 2 × 2 %atrices.

Pro!e that the follo ing f nction is an inner prod ct on M 22.

u4v ' ae 5 bf 5 cg 5 dh Deter%ine the inner prod ct of the %atrices .

==h g

f e

d c

bavu 4

olution$io% 1: u4v ' ae 5 bf 5 cg 5 dh ' ea 5 fb 5 gc 5 hd = v4

u

$io% &: 6et k be a scalar. Thenk u4v ' kae 5 kbf 5 kcg 5 kdh ' k (ae 5 bf 5 cg 5 dh) ' k u4

v *)01()90()2&()2(409

2

10

&2 =×+×+×−+×= −

−092

and10&2


22/79

!!

Example !.*8onsider the !ector space P n of polyno%ials of degree ≤ n. 6et f

and g be ele%ents of P n. Pro!e that the follo ing f nctiondefines an inner prod ct of P n.

Deter%ine the inner prod ct of polyno%ials

f ( x) ' x2 5 2 x 1 and g ( x) ' * x 5 1

∫ =1

0)()(g4 dx x g x f f

olution$io% 1: f g dx x f x g dx x g x f g f 4)()()()(4

1

0

1

0=== ∫ ∫

h g h f

dx xh x g dx xh x f

dx xh x g xh x f

dx xh x g x f h g f

44

)()();()(<

);()()()(<

)();()(


23/79


24/79

!+

orm of a ector

Definition 2.26et V be an inner prod ct space. The norm of a !ector v isdenoted >>v>> and it defined by

vv,v =

The nor% of a !ector in R n

can be e$pressed in ter%s of the dot prod ct as follo s

)444()444()()444(

2121

22121

nn

nn

x x x x x x x x x x x⋅=

++=

?enerali e this definition:The nor%s in general !ector space do not necessary ha!e geo%etricinterpretations4 b t are often i%portant in n %erical or/.


25/79

!,

Example !.+8onsider the !ector space P n of polyno%ials ith inner prod ct

The nor% of the f nction f generated by this inner prod ct is

Deter%ine the nor% of the f nction f ( x) ' x2 5 1.

∫ =1

0 )()(4 dx x g x f g f

∫ ==1

0

2);(


26/79

!4

Example !., 8onsider the !ector space M 22 of 2 × 2 %atrices. 6et u and v

defined as follo s be arbitrary 2 × 2 %atrices.

At is /no n that the f nction u4v ' ae 5 bf 5 cg 5 dh is aninner prod ct on M 22 by #$a%ple 2.

The nor% of the %atri$ is

==h g

f e

d c

bavu 4

22224 d cba +++== uuu


27/79

!8

Definition 2.!6et V be an inner prod ct space. The angle θ bet een t onon ero !ectors u and v in V is gi!en by

vuvu,=θ cos

The dot prod ct in R n as sed to define angle bet een !ectors.The angle θ bet een !ectors u and v in R n is defined by

( )cosvuvu ⋅=θ

An%le #etween two vectors


28/79

!9

An%le #etween two vectors

In " n #e first prove C$ ine%ualit&

' then define cos(

In " 2 #e first define cos(' then prove C$ine%ualit&


29/79

!=

Example !.48onsider the inner prod ct space P n of polyno%ials ith inner

prod ctThe angle bet een t o non ero f nctions f and g is gi!en by

Deter%ine the cosine of the angle bet een the f nctions f ( x) ' x2 and g ( x) ' & x

∫ =1

0 )()(4 dx x g x f g f

g f

dx x g x f

g f

g f

)()(

4cos

1

0∫ ==θ

olution =e first co%p te >> f >> and >> g >>.

&;&


30/79

*2

Example !.8 8onsider the !ector space M 22 of 2 × 2 %atrices. 6et u and v

defined as follo s be arbitrary 2 × 2 %atrices.

At is /no n that the f nction u4v ' ae 5 bf 5 cg 5 dh is aninner prod ct on M 22 by #$a%ple 2.

The nor% of the %atri$ is

The angle bet een u and v is

==h g

f e

d c

bavu 4

22224 d cba +++== uuu

22222222

4cos

h g f ed cba

dhcg bf ae

+++++++++==

vu

vuθ


31/79

*1

"rtho%onal ectorsDef 2.4. 6et V be an inner prod ct space. T o non ero !ectors u and v in V are said to be orthogonal if

04 =vu

Example 2.8

Bho that the f nctions f ( x) ' & x 2 and g ( x) ' x are orthogonal

in P n ith inner prod ct .)()(41

0∫ = dx x g x f g f olution

0;


32/79

*!

Jistance

Definition 2.)6et V be an inner prod ct space ith !ector nor% defined by

The distance bet een t o !ectors (points) u and v is definedd (u4v) and is defined by

vv,v =

)4( )4( vuvuvuvu −−=−=d

s for nor%4 the concept of distance ill not ha!e direct

geo%etrical interpretation. At is ho e!er4 sef l in n %erical%athe%atics to be able to disc ss ho far apart !ario sf nctions are .


33/79

**

Example !.98onsider the inner prod ct space P n of polyno%ials disc ssed

earlier. Deter%ine hich of the f nctions g ( x) ' x2

& x 5 or h( x)' x2 5 * is closed to f ( x) ' x2.olution

1&)&(&4&4);4(<1

0

22 =−=−−=−−= ∫ dx x x x g f g f g f d 1)*(*4*4);4(<

1

0

22 =−=−−=−−= ∫ dxh f h f h f d Th sThe distance bet een f and h is *4 as e %ight s spect4 g is closer

than h to f.

.*)4(and1&)4( == h f d g f d

* ' S h idt " th % li( ti


34/79

*+

*.'ram-Schmidt "rtho%onali(ation'iven a vector s 6 any vector v in an inner product space V can #e decomposed as

( ) proj proj= + −s sv v v v CC ⊥= +v v CC 0⊥× =v vwhereJefinition *.1 ; )utually "rtho%onal ectors

ectors v 16 K6 v k ∈ are mutually ortho%onal if v i Lv j 2

∀

i≠

j Theorem *.1 ;

A set of mutually ortho%onal non-(ero vectors islinearly independent.Proof ;

i ii

c =∑ v 0 0 j i i j j ji

c c=× =×∑v v v v

c j ' 0 ∀ j*+


35/79

*,

8orollary .&.1 :

set of k % t ally orthogonal non ero !ectors in Vk

is a basis forthe space.

Definition &.&: Orthonor%al Easis n orthonor%al basis for a !ector space is a basis of % t ally

orthogonal !ectors of nit length.

Definition &.2 : Orthogonal Easis n orthogonal basis for a !ector space is a basis of % t ally

orthogonal !ectors.

*.'ram-Schmidt "rtho%onali(ation

Definition &.* : Orthogonal 8o%ple%ent

The orthogonal co%ple%ent of a s bspace M of * is M ⊥ ' F v ∈ * >

v is perpendic lar to all !ectors in M G ( read H M perpI ).

The orthogonal projection proj M (v ) of a !ector is its projection into

M alon M ⊥ .


36/79

*4

6e%%a &.1 :

6et M be a s bspace of n. Then M ⊥ is also a s bspace and n. ' M ⊕ M ⊥ .

Jence4 ∀ v∈ n.4 v − proj M (v) is perpendic lar to all !ectors in M .

Proof : 8onstr ct bases sing ?-B orthogonali ation.

Theore% &.2 :

6et v be a !ector inn

and let M be a s bspace ofn.

ith basis β1 4 K4 βk .AfA is the %atri$ hose col %ns are the βLs then

proj M (v ) ' c1β1 5 K5 ck βk

here the coefficients ci are the entries of the !ector (A T A)-1 AT v. That is4

proj M (v ) ' A (A T A)−1 AT v.

Proof : ( ) M proj M ∈v

T T =A A c A v

here c is a col %n !ector

( )T = −0 A v Ac

( ) M proj =v A c

Ey le%%a&.M4

( ) 1T T −=c A A A v*4


37/79

*8

Anterpretation of Theore% &.2:

Af ' β1 4 K4 βk is an orthonor%al basis4 then A T A ' .

An hich case4 proj M (v ) ' A (A T A)−1 AT v ' A A T v.

( ) ( )1

1

T

M k T

k

proj

÷= × ÷ ÷

β

vββv

β

L M ( )1

1

T

k T

k

× ÷= ÷ ÷×

βv

ββ

βv

L M ( )1

1 k

k

v

v

÷= ÷ ÷

ββ L M

B

1

k

j j j

v=

=∑

βith j jv = ×βv

An partic lar4 if ' k 4 then A ' A T ' .

An case is not orthonor%al4 the tas/ is to find ! s.t. " ' A! and B T B ' .

( ) ( )T = A! A! T T = ! A A! ( )1 1T T − −=A A ! ! ( )1T −= !!

Jence

( )1T T −=!! A A

( ) T M proj =v "" v ( ) T = A! A! v T T = A!! A v ( )

1T T −= A A A A v

*8


38/79

*9

#$a%ple &.1 :

To orthogonally project1

1

1

÷= − ÷ ÷

v 0

x

P y x z

z

÷= + = ÷ ÷

1 0

0 1

1 0

÷= ÷ ÷−

A

into s bspace

7ro%1 0

0 1 4

1 0

P x y x y ÷ ÷= + ∈ ÷ ÷ ÷ ÷ −

R e get

( )1

1 01C 2 0 1 0 1

0 10 1 0 1 0

1 0

T T − − ÷= ÷ ÷ ÷ ÷−

A A A A

1 01 0 1

0 10 1 01 0

T

− ÷= ÷ ÷ ÷−

A A2 0

0 1

= ÷ ( )1 1C 2 0

0 1T −

= ÷ A A

1 01C 2 0 1C 2

0 10 1 0

1 0

− ÷= ÷ ÷ ÷−

1C 2 0 1C 2

0 1 0

1C 2 0 1C 2

− ÷= ÷ ÷−

( )1C 2 0 1C 2 1

0 1 0 1

1C 2 0 1C 2 1 P proj

− ÷ ÷= − ÷ ÷ ÷ ÷−

v

0

1

0

÷= − ÷ ÷

*9


39/79

*=

Exercises *.

1. Perform the 'ram-Schmidt process on this #asis for , & 4

2 1 0

2 4 0 4 &

2 1 1

÷ ÷ ÷ ÷ ÷ ÷ ÷ ÷ ÷−

2. Show that the columns of an n×n matrix form anorthonormal set if and only if the inverse of the matrix is itstranspose. Produce such a matrix.

*=


40/79

+2

+ Projection Into a Su#spaceJefinition *.1 ; or any direct sum V M⊕ N and any v ∈ V

such that v m / n with m ∈ M and n ∈ N .The projection of v into M alon% N is defined as E5v7 proj M 6N 5v 7 m

eminder ; M M N need not #e ortho%onal.There need not

even #e an inner product defined.

+2

⊕

Theorem*.1; Show that 5i7 E is linear and 5ii7 E ! E.

Theorem*.!; &et E; > is linear and E! E then65i7 E5u7 u for any u N ImE. 5ii7 is the direct sum of the

ima%e and Bernel of E. i.e.6 ImE DerE. 5iii7 E is theprojection of into ImE6 its ima%e alon% DerE.

Theorem*.!; &et E; > is linear and E! E then65i7 E5u7 u for any u N ImE. 5ii7 is the direct sum of the

ima%e and Bernel of E. i.e.6 ImE DerE. 5iii7 E is theprojection of into ImE6 its ima%e alon% DerE.


41/79

+1

Projection and )atrices

O &et &5 7 & & is a linear map #etween and itself3. &5 7is a vector space under function addition and scalarmultiplication.

O dim5&7 n! if d5 7 n.O If we fix a #asis in 6 there arises a one to one

correspondence #etween & and set of all matrices oforder n 5the last set is also a vector space withdimension n ! 7. The result implies that matrices identifylinear operators.


42/79

+!

"rtho%onal Projection

"

Q 516!7

5!617R

two vector spaces 16 ! #y multiplyin% the vectors and#y B where B .

1 B and ! B

2

1

1

2

∈

+!

2

1

1

2


43/79

+*

*rthogonal +ro,ection

ow6 we can find the vector space ! as 1 !&et #e any vector of ! and B 16 B! then we can

write B

1 /B

!

Therefore6

⊕

2

1

x

x

2

1

x

x

2

1

1

2

∈

+*

2

1

x

x

12

21

2

1

k

k

2

1

k

k 1

12

21 −

2

1

k

k

−

+−

21

21

&1

&2

&2

&1

x x

x x

2

1

x

x


44/79

++

"rtho%onal Projectionow6 let P #e a projection matrix and x #e a vector in ! 6

then a projection from ! to 1 is %iven #y6

Px B 1

++

2

1-1F* x1 /!F* x ! U 2

1-

−

−

&

*

&

2&

2

&

1

2

1

x

x

Ex. 17 ChecB that P is idempotent #ut not symmetric. :hy<

!7 Prove that if the second vector is 6 P will #e then

idempotent as well as symmetric

−12


45/79

+,

)eanin% of P nVn xnV1O Case 1; P nVn is sin%ular #ut not idempotent

≠21 1

, ( ) 1,2 2

P rank P P P

1 13

2 2

The whole space6 n is mapped to the column space ofPnVn 6 an improper su#space of n .

An vector of the su#space may mapped to another vector ofthe Su#space.6

1 12 21 1

1 2

2 2

1 1 1( )

2 2 2

P x x x x

x x


46/79

+4

)eanin% of P nVn xnV1

O Case !; PnVn

is sin%ular and idempotent5 asymmetric7

21 0

, ( ) 11 0

P rank P and P P

¬11

2

1

1Px

x x

x 3 1

32 1

→

Px is not ortho%onal to x-Px

2 2

2 2P

The whole space6 n

is mapped to the columnspace of PnVn 6 an improper su#space of n .

An vector of the su#space is mapped to the samevector of the Su#space.6


47/79

+8


O Case *; P nVn is sin%ular and idempotent5 symmetric7

)eanin%; The whole space6 n is mapped to the column space ofP nVn 6 an improper su#space of n . An vector of the su#space ismapped to the same vector of the Su#space. It is ortho%onal

projection6 That is6 the su#space is to its complement. or example6

2

1

2

1

2

1

2

1

Px 5x1/x ! 7 1/ 2

1/ 2 Px is ortho%onal to x-Px


48/79

+9


O Case +; P nVn is non-sin%ular and non-ortho%onal )eanin%; The whole space6 n is mapped to the column space of

P nVn 6 same as n . The mappin% is one-to-one and onto.:e havenow columns of P nVn as a new 5o#li$ue7 #asis in place of standard#asis. An%les #etween vectors and len%th of vectors are notpreserved. or example6

1 22 1

1, x y Px y x P y → →


49/79

+=

)eanin% of P nVn xnV1O Case ; P nVn is non-sin%ular and ortho%onal

)eanin%; The whole space6 n is mapped to the column space ofP nVn 6 same as n . The mappin% is one-to-one and onto.:ehave now columns of P nVn as a new 5ortho%onal7 #asis in place

of standard #asis. An%les #etween vectors and len%th of vectorsare preserved. :e have only a rotation of axes. or example6

2

1-

2

1

2

1

2

1 rom a symmetricmatrix we havealways such a Pof its nindependentei%en vectors

rom a symmetricmatrix we have alwaya symmetricidempotent P of itsr5Wn 7independentei%en vectors

P j i Th I Xil# S


50/79

,2

Projection Theorem In a Xil#ert Space

&et #e a closed su#space of a Xil#ert space6 . Thereexists a uni$ue pair of mappin%s P;X → and Y; → suchthat x Px/Yx for all xN . P and Y have the followin%properties;

i7 x N x x6 Yx 2

ii7 x N T Px 26 Yx x

ii7 Px is closest vector in to x.

iv7 Yx is closest vector in to x

v7 Px! / Yx ! x!

vi7 P and Y are linear maps and P ! P6 Y ! Y

⇒

⇒

⇒

inite dimensional spaces are always closed

1


51/79

,1

The common characteristic 5structure7 amon% thefollowin% statistical methods<

1. Principal Components Analysis!. 5 id%e 7 re%ression*. isher discriminant analysis+. Canonical correlation analysis,.Sin%ular value decomposition4. Independent component analysis

Applications

We consider linear co binations of inp!t vector" ( ) T f x # x=:e maBe use concepts of len%th and dot productavaila#le in Euclidean space.


52/79

,!

:hat is feature reduction<

d $ ℜ∈ pd T % ×ℜ∈

p & ℜ∈

d T d p

& %$ & % ℜ∈=→ℜ∈ ×

:

&inear transformation

"ri%inal data reduced data

Ji i li d i


53/79

,*

Jimensionality eduction

O "ne approach to deal with hi%h dimensional data is #yreducin% their dimensionality.

O Project hi%h dimensional data onto a lower dimensionalsu#-space usin% linear or non-linear transformations.


54/79

,+

Principal Component Analysis 5PCA7

1 1 2 2

1 2

" ...here 4 4...4 isa basein the -di%ensionals b-space (NO3)

' '

'

x b ! b ! b !! ! ! ' = + + +

" x x=

1 1 2 2

1 2

...here 4 4...4 isa basein the original 3-di%ensionalspace

( (

n

x a v a v a vv v v

= + + +

O ind a #asis in a low dimensional su#-space;

Z Approximate vectors #y projectin% them in a low dimensional su#-space;

517 "ri%inal space representation;

5!7 &ower-dimensional su#-space representation;

O Note if D 6 then

P i i l C t A l i 5PCA7


55/79

,,

Principal Component Analysis 5PCA7O Information loss

ZJimensionality reduction implies information loss [[ Z PCA preserves as much information as possi#le;

O :hat is the ?#est@ lower dimensional su#-space< The ?#est@ low-dimensional space is centered at the sample mean and has directions determined #y the ?#est@ ei%envectors of the covariance matrix of the data x.

Z y ?#est@ ei%envectors we mean those correspondin% to the largest ei%envalues 5 i.e.6 principal components/ 7.

Z Since the covariance matrix is real and symmetric6 these ei%envectors are ortho%onal and form a set of #asis vectors.

"%in >> >> (reconstr ction error) x x−

ZSin%ular alue


56/79

,4

Principal Component Analysis 5PCA7O )ethodolo%y

Z Suppose x 16 x ! 6 ...6 x M are N x 1 vectors


57/79

,8

Principal Component Analysis 5PCA7O )ethodolo%y Z cont.

( )T i ib ! x x= −


58/79

,9

Principal Component Analysis 5PCA7O &inear transformation implied #y PCA

Z The linear transformation R N

→ R !

that performs the dimensionality reduction is;

P i i l C t A l i


59/79

,=

Principal Component Analysis5PCA7

O Ei%envalue spectrum

# i$ # %


60/79

42

Principal Com(ponent Analysis 5PCA7O :hat is the error due to dimensionality reduction<

O It can #e shown that error due to dimensionality reduction is e$ual to;

∑

(

k ii

(

iie

11

2


61/79

41

Principal Component Analysis 5PCA7O Standardi(ation

Z The principal components are dependent on the "nits used tomeasure the ori%inal varia#les as well as on the range of values theyassume.

Z :e should always standardi(e the data prior to usin% PCA. Z A common standardi(ation method is to transform all the data to

have (ero mean and unit standard deviation;



62/79

4!


O Case Study ; Ei%enfaces for ace JetectionF eco%nition

Z ). TurB6 A. Pentland6 \Ei%enfaces for eco%nition\6 #o"rnal of $ogniti%eNe"roscience 6 vol. *6 no. 16 pp. 81-946 1==1.

O 0ace "ecognition

Z The simplest approach is to thinB of it as a templatematchin% pro#lem

Z Pro#lems arise when performin% reco%nition in ahi%h-dimensional space.

Z Si%nificant improvements can #e achieved #y firstmappin% the data into a lo&er di'ensionalityspace.

Z Xow to find this lower-dimensional space<


63/79

4*

Principal Component Analysis 5PCA7O )ain idea #ehind ei%enfaces

a!erage face


64/79

4+

Principal Component Analysis 5PCA7O Computation of the ei%enfaces Z cont.

i

)

i n d t h a t t h i s i s n

or m

al i (

e d ..


65/79

4,

Principal Component Analysis 5PCA7O Computation of the ei%enfaces Z cont.



66/79

44

Principal Component Analysis 5PCA7O epresentin% faces onto this #asis



67/79

48


O epresentin% faces onto this #asis Z cont.



68/79

49

Principal Component Analysis 5PCA7O ace eco%nition ]sin% Ei%enfaces


69/79

4=

Principal Component Analysis 5PCA7O ace eco%nition ]sin% Ei%enfaces Z cont.

Z The distance e r is called distance &ithin the face space 5difs 7

Z Comment; we can use the common Euclidean distance to compute e r 6however6 it has #een reported that the Mahalano(is distance performs #etter;



70/79

82

Principal Component Analysis 5PCA7O ace Jetection ]sin% Ei%enfaces

l l


71/79

81

Principal Component Analysis 5PCA7O ace Jetection ]sin% Ei%enfaces Z cont.

Principal Component Analysis


72/79

8!

Principal Component Analysis5PCA7O econstruction

of faces and non-faces

Principal Component Analysis


73/79

8*

Principal Component Analysis5PCA7

O Applications

Z ace detection6 tracBin%6 and reco%nitiondffs

Principal Components


74/79

8+

Principal Components Analysis

So6 principal components are %iven #y;

( 1 " 11 x 1 / " 1! x ! / ... / " 1 x

( ! " !1 x 1 / " !! x ! / ... / " ! x ...( a 1 x 1 / a ! x ! / ... / a x

x j Gs are standardi(ed if correlation 'atrix is used5mean 2.26 SJ 1.27


75/79

8,


core of i th unit on j th principalcomponent

( i,j " j 1 x i 1 / " j ! x i ! / ... / " jN x iN


76/79

84

PCA Scores

+.2 +., ,.2 ,., 4.2!

*

+

,

xi)

xi*

bi+* bi+)


77/79

88


Amount of variance accounted for #y;1st principal component6 ^16 1st eigenvalue

!nd principal component6 ^!6 !nd eigenvalue

...

^1 _ ^ ! _ ^* _ ^ + _ ...

Avera%e ̂j 1 5correlation matrix7

Principal Components Analysis;


78/79

89

Principal Components Analysis;Ei%envalues

+.2 +., ,.2 ,., 4.2!

*

+

,

1 2

1


79/79

han3 &ou

Examples Done on Orthogonal Projection

Documents