5 - 1 3.8 Inner Product Spaces Euclidean n-space: Euclidean n-space: vector lengthdot productEuclidean n-space R n was defined to be the set of all ordered.

5 - 1

3.8 Inner Product Spaces3.8 Inner Product Spaces

Euclidean Euclidean nn-space:-space:

Rn was defined to be the set of all ordered n-tuples of real

numbers. When Rn is combined with the standard operations of

vector addition, scalar multiplication, vector lengthvector length, and the dot dot

productproduct, the resulting vector space is called Euclidean Euclidean nn-space-space.

1 1 2 2u v n nu v u v u v L

The dot product of two vectors is defined to be

The definitions of the vector length and the dot product are needed to provide the metric conceptthe metric concept for the vector space.

5 - 2

(1)

(2)

(3)

(4) and if and only if

〉〈〉〈 uvvu ,,

〉〈〉〈〉〈 wuvuwvu ,,, 〉〈〉〈 vuvu ,, cc

0, 〉〈 vv 0, 〉〈 vv 0v

Axioms of inner product:Axioms of inner product:

Let u, v, and w be vectors in a vector space V, and let c be

any scalar. An inner product on An inner product on VV is a function that is a function that

associates a real number <u, v> with each pair of vectors associates a real number <u, v> with each pair of vectors

u and vu and v and satisfies the following axioms.

5 - 3

Note:

V

Rn

space for vectorproduct inner general,

)for product inner Euclidean (productdot

vu

vu

Note:

A vector space V with an inner product is called an inner

product space.

, ,V Vector space:

Inner product space: , , , ,V

5 - 4

Ex: (A different inner product for Rn)

Show that the function defines an inner product on R2, where and .

2211 2, vuvu 〉〈 vu),( ),( 2121 vvuu vu

Sol: 〉〈〉〈 uvvu ,22, )( 22112211 uvuvvuvua

〉〈〉〈

〉〈

wuvu

wvu

,,

)2()2(

22

)(2)(,

22112211

22221111

222111

wuwuvuvu

wuvuwuvu

wvuwvu),( )( 21 wwb w

〉〈〉〈 vuvu ,)(2)()2(, )( 22112211 cvcuvcuvuvuccc

02, )( 22

21 vvd 〉〈 vv

)0(0020, 212

22

1 vvv vvvv〉〈

5 - 5

Ex: (A function that is not an inner product)

Show that the following function is not an inner product on R3.

332211 2 vuvuvu 〉〈 vu

Sol:

Let )1,2,1(v

06)1)(1()2)(2(2)1)(1(,Then vv

Axiom 4 is not satisfied.

Thus this function is not an inner product on R3.

5 - 6

Ex: (A function that is not an inner product)

Show that the following function is not an inner product on R3.

332211 2 vuvuvu 〉〈 vu

Sol:

Let )1,2,1(v

06)1)(1()2)(2(2)1)(1(,Then vv

Axiom 4 is not satisfied.

Thus this function is not an inner product on R3.

5 - 7

Norm (length) of u:

〉〈 uuu ,|||| 〉〈 uuu ,|||| 2

(5) If A and B are two matrices, an inner product can be <A,B>=Tr(A†B), where † is the transpose complex conjugate of the matrix and Tr means the trace. Therefore

For a norm, there are many possibilities.

5 - 8

For a norm, there are many possibilities.

There is an example in criminal law in which the distinctions between some of these norms has very practical consequences. If you’re caught selling drugs in New York there is a longer sentence if your sale is within 1000 feet of a school. If you are an attorney defending someone accused of this crime, which of the norms would you argue for? The legislators didn’t know linear algebra, so they didn’t specify which norm they intended. The prosecuting attorney argued for norm #1, “as the crow flies.” The defense argued that “crows don’t sell drugs” and humans move along city streets, so norm #2 is more appropriate.

The New York Court of Appeals decided that the Pythagorean norm (#1) is the appropriate one and they rejected the use of the pedestrian norm that the defendant advocated (#2).

5 - 9

DistanceDistance between u and v:

vuvuvuvu ,||||),(d

AngleAngle between two nonzero vectors u and v:

0,||||||||

,cos

vu

vu 〉〈

u and v are orthogonal if .

0, 〉〈 vu

OrthogonalOrthogonal: )( vu

Note:Note: If , then v is called a unit vectorunit vector. 1|||| v

0

1

v

v gNormalizin

v

v (the unit vector in the direction of v)

5 - 10

Properties of norm:Properties of norm:

(1)

(2) if and only if

(3)

Properties of distance:Properties of distance:

(1)

(2) if and only if

(3)

0|||| u

0|||| u 0u

|||||||||| uu cc

0),( vud

0),( vud vu

),(),( uvvu dd

5 - 11

Ex: (Finding inner product)

)(in spolynomial be24)(,21)(Let 222 xPxxxqxxp

0 0 1 1, n np q a b a b a b L is an inner product

( ) , ?a p q 〈 〉 ( ) || || ?b q ( ) ( , ) ?c d p q

Sol:Sol:( ) , (1)(4) (0)( 2) ( 2)(1) 2a p q 〈 〉

2 2 2( ) || || , 4 ( 2) 1 21b q q q 〈 〉2

2 2 2

( ) 3 2 3

( , ) || || ,

( 3) 2 ( 3) 22

c p q x x

d p q p q p q p q

Q

2: (1,0, 2) (4, 2,1) {1, , }Note p and q with basis x x

5 - 12

ThmThm 3.21 3.21：

Let u and v be vectors in an inner product space V.

(1) Cauchy-Schwarz inequalityCauchy-Schwarz inequality:

(2) Triangle inequalityTriangle inequality:

(3) Pythagorean theoremPythagorean theorem :

u and v are orthogonal if and only if

||u v|| ||u|| ||v||

222 |||||||||||| vuvu

| u , v | ||u|| ||v||〈 〉

5 - 13

NoteNote:We can solve for this coefficient by noting that because is orthogonal to a scalar multiple of , it must be orthogonal to itself. Therefore, the consequent fact that the dot product is zero, giving that

Orthogonal projectionsOrthogonal projections in inner product spaces:

Let u and v be two vectors in an inner product space V,

such that . Then the orthogonal projection of u onto

v is given by

0v

,proj ,

,

v

u vu u v v

v vv

u

s

sr

s( Proj )v v s rr r r

g

sProjv v rr r

proj ,s v v s s rr r r r

sr

sr

5 - 14

Ex: (Finding an orthogonal projection in R3)

Use the Euclidean inner product in R3 to find the

orthogonal projection of u=(6, 2, 4) onto v=(1, 2, 0).

Sol:u , v (6)(1) (2)(2) (4)(0) 10 Q

2 2 2v , v 1 2 0 5

proj 10v 5

u vu v (1 , 2 , 0) (2 , 4 , 0)

v v

5 - 15

ThmThm 3.22 3.22: (Orthogonal projection and distance)

Let v and s be two vectors in an inner product space V,

such that . Then 0s

proj,

( , ) ( , ) , ,s

v sd v v d v cs c

s s

r

r rr r r r

r r

5 - 16

3.93.9 Orthonormal Bases: Gram-Schmidt ProcessOrthonormal Bases: Gram-Schmidt Process OrthogonalOrthogonal:

A set S of vectors in an inner product space V is called an

orthogonal setorthogonal set if every pair of vectors in the set is

orthogonal.

OrthonormalOrthonormal:

An orthogonal set in which each vector is a unit vectorunit vector is

called orthonormal.

1 2

1v , v , , v v , v

0n i j

i jS V

i j

1 2v , v , , v v , v 0n i jS V L ji

5 - 17

The standard basis is orthonormal.

Ex: (An orthonormal basis for )

In , with the inner product

)(3 xP

0 0 1 1 2 2,p q a b a b a b

21 2 3{1, , } { , , }B x x v v v

3( )P x

Sol:Sol: ,001 21 xx v ,00 2

2 xx v ,00 23 xx v

1 2

1 3

2 3

, (1)(0) (0)(1) (0)(0) 0,

, (1)(0) (0)(0) (0)(1) 0,

, (0)(0) (1)(0) (0)(1) 0

v v

v v

v v

Then 1110000

,1001100

,1000011

333

222

111

v,vv

v,vv

v,vv

Thus, B is an orthonormal basis for .)(3 xP

5 - 18

Thm 3.23:Thm 3.23: (Orthogonal sets are linearly independentOrthogonal sets are linearly independent)

If is an orthogonal setorthogonal set of nonzero vectors

in an inner product space V, then S is linearly independentlinearly independent.

1 2v , v , , vnS L

Pf:

S is an orthogonal set of nonzero vectors

0and0i.e. iiji ji v,vv,v

1 1 2 2

1 1 2 2

Let v v v 0

v v v , v 0, v 0n n

n n i i

c c c

c c c i

L

L

1 1 2 2 v , v v , v v , v v , v

0i i i i i n n ic c c c

L L

v , v 0 0 is linearly independent.i i ic i S Q

5 - 19

Ex: (Using orthogonality to test for a basis)

Show that the following set is a basis for .4R

)}1,1,2,1(,)1,2,0,1(,)1,0,0,1(,)2,2,3,2{(4321

S

vvvv

Sol:

: nonzero vectors

02262

02402

02002

41

31

21

vv

vv

vv

4321 ,,, vvvv

01201

01001

01001

43

42

32

vv

vv

vv

.orthogonal is S4for basis a is RS

5 - 20

Thm 3.24:Thm 3.24: (Coordinates relative to an orthonormal basis)

If is an orthonormal basisorthonormal basis for an inner

product space V, then the coordinate representation of a vector

w with respect to B is

1 2{v , v , , v }nB L

1 2{v , v , , v }nB Q L is orthonormal

ji

jiji

0

1, vv

Vw

1 1 2 2w v v vn nk k k L (unique representation)

Pf:Pf:

is a basis for V1 2{v , v , , v }nB L

1 1 2 2[w] w , v v w , v v w , v vB n n L

5 - 21

1 1 2 2

1 1

w , v ( v v v ) , v

v , v v , v v , vi n n i

i i i i n n i

i

k k k

k k k

k i

L

L L

1 1 2 2w w, v v w, v v w, v vn n L

1

2

w, v

w, vw

w, v

B

n

M

5 - 22

Ex: (Representing vectors relative to an orthonormal basis)

Find the coordinates of w = (5, -5, 2) relative to the following

orthonormal basis for .

3 34 41 2 3 5 5 5 5{ , , } {( , , 0) ,( , , 0) , (0 , 0 , 1)}B v v v

3R

Sol:

2)1,0,0()2,5,5(,

7)0,,()2,5,5(,

1)0,,()2,5,5(,

33

53

54

22

54

53

11

vwvw

vwvw

vwvw

2

7

1

][ Bw

5 - 23

Gram-Schmidt orthonormalization process:Gram-Schmidt orthonormalization process:

is a basis for an inner product space V 1 2{u , u , , u }nB L

11Let uv })({1 1vw span

}),({2 21 vvw span

1 2' {v , v , , v }nB L

1 2

1 2

vv v'' { , , , }

v v vn

n

B L

is an orthogonal basis.

is an orthonormal basis.

1

1

W1

v , vv u proj u u v

v , vn

nn i

n n n n ii i i

M2

3 1 3 23 3 W 3 3 1 2

1 1 2 2

u , v u , vv u proj u u v v

v , v v , v

111

122222 〉〈

〉〈proj

1v

v,v

v,uuuuv W

5 - 24

Sol: )0,1,1(11 uv

)2,0,0()0,2

1,

2

1(

2/1

2/1)0,1,1(

2

1)2,1,0(

222

231

11

1333

vvv

vuv

vv

vuuv

Ex: (Applying the Gram-Schmidt orthonormalization process)

Apply the Gram-Schmidt process to the following basis.

)}2,1,0(,)0,2,1(,)0,1,1{(321

B

uuu

)0,2

1,

2

1()0,1,1(

2

3)0,2,1(1

11

1222

vvv

vuuv

5 - 25

}2) 0, (0, 0), , 2

1 ,

2

1( 0), 1, (1,{},,{' 321

vvvB

Orthogonal basis

}1) 0, (0, 0), , 2

1 ,

2

1( 0), ,

2

1 ,

2

1({},,{''

3

3

2

2

v

v

v

v

v

v

1

1B

Orthonormal basis

5 - 26

Ex: Find an orthonormal basis for the solution spacesolution space of the

homogeneous system of linear equations.

0622

07

4321

421

xxxx

xxx

Sol:

08210

01201

06212

07011 .. EJG

1

21 2

3

4

2 2 1

2 8 2 8

1 0

0 1

x s t

x s ts t sv tv

x s

x t

r r

5 - 27

Thus one basis for the solution space is

)}1,0,8,1(,)0,1,2,2{(},{ 21 vvB

1 ,2 ,4 ,3

0 1, 2, ,2 9

181 0, 8, 1,

,

,

0 1, 2, ,2

111

1222

11

vvv

vuuv

uv

1,2,4,3 0,1,2,2' B (orthogonal basis)

30

1,

30

2,

30

4,

30

3 , 0,

3

1,

3

2,

3

2''B

(orthonormal basis)

5 - 28

3.10 Mathematical Models and Least-Squares Analysis3.10 Mathematical Models and Least-Squares Analysis

Let WW be a subspacesubspace of an inner product space VV.

(a) A vector u in V is said to orthogonal to W, if u is

orthogonal to every vector in W.

(b) The set of all vectors in V that are orthogonal to W is

called the orthogonal complement of W.

(read “ perp”)

} ,0,|{ WVW wwvv

W W

Orthogonal complementOrthogonal complement of W:

5 - 29

Direct sumDirect sum:

Let and be two subspaces of . If each vector

can be uniquely written as a sum of a vector from

and a vector from , , then is the

direct sum of and , and you can write

1W 2W nR nRx

1W1w

2W2w21 wwx nR

21 WWRn

ThmThm 3.25 3.25: (Properties of orthogonal subspaces) Let W be a subspace of Rn. Then the following properties

are true.

(1)

(2)

(3)

nWW )dim()dim( WWRn

WW )(

1W 2W

5 - 30

Thm 3.26Thm 3.26: (Projection onto a subspace)

If is an orthonormal basis for the

subspace W of V, and for , then

1 2{u , u , , u }tL

Vv

vv W

proj

||

proj 1 1 2 2v v , u u v , u u v , u uW t t L

proj v, 0 iWu i Q

5 - 31

Ex: (Projection onto a subspace)

3 ,1 ,1 ,0 ,0 ,2 ,1 ,3 ,0 21 vww

Find the projection of the vector v onto the subspace W.

:, 21 ww

Sol:

an orthogonal basis for W

:0,0,1),10

1,

10

3,0( , ,

2

2

1

121

w

w

w

wuu

an orthonormal basis for W

}),({ 21 wwspanW

5 - 32

Scientists are often presented with a system that has no solution and they must find an answer that is as close as possible to being an answer.

Suppose that we have a coin to use in flipping and this coin hassome proportion m of heads to total flips.

Because of randomness, we do not find the exact proportion with this sample

The vector of experimental data {16, 34, 51} is not in the subspace of solutions.

Fitting by Least-SquaresFitting by Least-Squares

5 - 33

However, we want to find the m that most nearly works. An orthogonal projection of the data vector into the line subspace gives our best guess.

The estimate (m = 7110/12600 ~ 0.56) is a bit high than 1/2 but not much, so probably the penny is fair enough. The line with the slope m= 0.56 is called the line of best fitline of best fit for this data.

Minimizing the distance between the given vector and the vector used as the left-hand side minimizes the total of these vertical lengths. We say that the line has been obtained through fitting by fitting by least-squaresleast-squares.

5 - 34

The different denominations of U.S. money have different averagetimes in circulation

The linear system with equations has no solution, but we can use orthogonal projection to find a best approximation.

1 1 1.5

1 5 2

1 10 3

1 20 5

1 50 9

1 100 20

bAx v

m

tr r

5 - 35

The method on the Projection into a Subspace says that coefficients b and m so that the linear combination of the columns of AA is as close as possible to the vector are the entries of

Some calculation gives an intercept of b b = 1= 1..0505 and a slope of m m = 0= 0..1818.

vr

1( )T Tbx A A A v

m

r r

5 - 36

Thm 3.27Thm 3.27: (Orthogonal projection and distance)

Let SS be a subspace of an inner product space V, and .

Then for all ,

Vv

s S proj vSs

proj||v v|| ||v ||S s

o projr ||v v|| min ||v ||S s

( is the best approximation to v from SS)proj vS

5 - 37

Pf:Pf:proj projv (v v) ( v )S Ss s

proj proj(v v) ( v )S S s

By the Pythagorean theorem

proj proj2 2 2||v || ||v v|| || v ||S Ss s

proj projv v 0S Ss s

proj2 2||v || ||v v||Ss

proj||v v|| ||v ||S s

5 - 38

For a given mnmn matrix [AAmnmn]: The spaces NS(A) and RS(A) are orthogonal

complements of each other within Rn. This means that any vector from NS(A) is

orthogonal to any vector from CS(AT), and the vectors in these two spaces span

Rn, i.e., ( ) ( )n TR NS A CS A

.

Fundamental subspaces of a matrixFundamental subspaces of a matrix

5 - 39

Thm 3.28Thm 3.28:

If A is an m×n matrix, then

(1)

(2)

(3)

(4)

( ( )) ( )

( ( )) ( )

CS A NS A

NS A CS A

( ( )) ( )

( ( )) ( )

CS A NS A

NS A CS A

( ) ( ) ( ) ( ( ))T m mCS A NS A R CS A NS A R

( ) ( ) ( ) ( ( ))T n T nCS A NS A R CS A CS A R

5 - 40

Ex: (Fundamental subspaces)

Find the four fundamental subspaces of the matrix.

000

000

100

021

A (reduced row-echelon form)

Sol: span 4( ) 1,0,0,0 0,1,0,0 is a subspace of CS A R

span 3( ) 1,2,0 0,0,1 is a subspace of CS A RS A R

span 3( ) 2,1,0 is a subspace of NS A R

5 - 41

span 4( ) 0,0,1,0 0,0,0,1 is a subspace of NS A R

1 0 0 0 1 0 0 0

2 0 0 0 0 1 0 0

0 1 0 0 0 0 0 0

A R

Check:

)())(( ANSACS

)())(( ANSACS

4)()( RANSACS T 3)()( RANSACS T

ts

5 - 42

Ex:

Let W is a subspace of R4 and .

(a) Find a basis for W

(b) Find a basis for the orthogonal complement of W.

)1 0, 0, 0,( ),0 1, 2, 1,( 21 ww

Sol:

1 2

1 0 1 0

2 0 0 1

1 0 0 0

0 1 0 0

w w

A R

(reduced row-echelon form)

}),({ 21 wwspanW

5 - 43

( )

1,2,1,0 , 0,0,0,1

a W CS A

is a basis for W

1

2

3

4

( )

2 2 1

1 2 1 0 1 0

0 0 0 1 0 1

0 0 0

2,1,0,0 1,0,1,0 is a basis for

b W CS A NS A

x s t

x sA s t

x t

x

W

Q

Notes:

4

4

(2)

)dim()dim()dim( (1)

RWW

RWW

5 - 44

Least-squares problem:Least-squares problem:

(A system of linear equations)

(1) When the system is consistent, we can use the Gaussian

elimination with back-substitution to solve for x

A x b11 mnnm

(2) When the system is consistent, how to find the “best possible”

solution of the system. That is, the value of x for which the

difference between Ax and b is small.

5 - 45

Least-squares solutionLeast-squares solution:

Given a system Ax = b of m linear equations in n unknowns,

the least squares problem is to find a vector x in Rn that

minimizes with respect to the Euclidean inner

product on Rn.

Such a vector is called a least-squares solutionleast-squares solution of Ax = b.

bx A

5 - 46

x

x ( ) ( is a subspace of )

( )

nm n

m

A M R

A CS A CS A R

Let W CS A

Q

1ˆ

x̂ b

ˆ(b x) ( Pr b)

ˆ( x) ( )

ˆb x ( ( )) ( )

ˆ(b x) 0

ˆi.e. x b ( ),

W

T T

W

A Proj

A b oj W

b A CS A

A CS A NS A

A A

A A A o x A A br A

(This is the solution of the normal system associated

with Ax = b)

5 - 47

Note: Note: The problem of finding the least-squares solutionthe least-squares solution of

is equal to the problem of finding an exact solution of the an exact solution of the

associated normal systemassociated normal system .

bx A

bx AAA ˆ

Thm:Thm:

If A is an m×n matrix with linearly independent column vectors,

then for every m×1 matrix b, the linear system Ax = b has a unique

least-squares solution. This solution is given by

Moreover, if W is the column space of A, then the orthogonal

projection of b on W is

1x ( ) bLS A A A

1b x bproj [ ( ) ]W AA A A A

5 - 48

Ex: (Solving the normal equations)

Find the least squares solution of the following system

and find the orthogonal projection of b on the column space of A.

5 - 49

Sol:

the associated normal system

5 - 50

the least squares solution of Ax = b

53

32

xLS

the orthogonal projection of b on the column space of A

165

3 8( ) 63

2 176

1 1

proj 1 2

1 3

b xCS A LSA