nt.tuwien.ac.at - Signal Processing 1 · 2019. 1. 23. · 2 Univ.-Prof. Dr.-Ing. Markus Rupp Learning goals Vector Spaces and Applications of Linear Algebra in Signal Processing (6Units,

Signal Processing 1

Signal Spaces

Univ.-Prof.,Dr.-Ing. Markus RuppWS 17/18

Th 14:00-15:30EI3A, Fr 8:45-10:15EI4

LVA 389.166

Last change: 20.11.2017

2Univ.-Prof. Dr.-Ing.

Markus Rupp

Learning goals Vector Spaces and Applications of Linear Algebra

in Signal Processing (6Units, Chapter 2) Metric spaces, sequences, Cauchy-sequences, supremum,

infimum, sparsity (Ch 2.1) Groups, Vector spaces, linear combination, linear

independence, basis and dimension, orthogonality,blind channel estimation (Ch 2.2, 2.7)

Norms and normed vector spaces (Ch 2.3-2.5) Applications of norms: robustness descriptions,

feedback systems with nonlinear elements: the small gain theorem

Inner vector products and inner product spaces, Hilbert and Banach spaces (Ch 2.8-2.9)

Induced norms, Cauchy-Schwarz inequality: matched filter and correlation coefficient, time-frequency uncertainty (Ch 2.6)

Sets, Spaces, and Vectors A set is a collection of distinct objects. A vector is an n-dimensional set of n ordered

elements. A space is a set with “some added structure”.


Markus Rupp


Markus Rupp

Example 2.1: Metric

Code bookCode word

1,0==⊂∈

BYBCx n

Decoder has to decide which one among the received y is allowed and most probable!


Markus Rupp

Example 2.1: Metric

Code Rate:

)(log2 Mk =# Info bits

# Code bits kn ≥

nM

nkR )(log2==

−),( kn Codexy

k2=


Markus Rupp

Example 2.1: Metric Decision of the „most probable symbol“

based on metric: Compare “distance” of received y to all possible,

allowed x. Symbol x with smallest (Hamming-) distance dH

is interpreted as correct.

( ) ∑ =⊕=

→×=n

i iiH

nnH

yxyxd

NBBdB

1

0

,

:,1,0


Markus Rupp

Example 2.2 We transfer two possible signal forms from two

different sensors: f(t) and g(t). The receiver has to find out whether the

distorted received signal form r(t) is closer to f(t) or to g(t).

We need a distance measure from r(t) to f(t) and from r(t) to g(t):

∫∫ −=−=b

a

b

a

dttgtrgrdvsdttftrfrd 22

22 )()(),(.)()(),(


Markus Rupp

Metric Spaces Consider a set X in R,C,N,Z,B… Definition 2.1: A metric

is a functional mapping to measure distances between objects/elements of the set X. In order to call such a distance a metric, the following properties need to be satisfied:

)(: 00 NRXXd +→×

Xzyxzydyxdzxdyxyxd

xydyxdyxd

∈+≤==

=≥

,, allfor );,(),(),()3 ifonly and if ,0),()2

),(),()10),()0

Metric Note that 0) follows from 1), 2), and

3) since:


Markus Rupp

0),( :thus3) and 1) todue

),(2),(),(),(0:have wethen

)2 todue 0),(

≥

=+≤=

=

yxd

yxdxydyxdxxd

xxd


Markus Rupp

Metric Spaces Example 2.3: Consider the following metric

d1(x,y)

defined over the vectors x,y. This metric is called l1 metric. (Manhattan distance)

Example 2.4: Consider the following metric dp(x,y)

( ) ∑ =

+

−=

→×n

i ii

nn

yxyxd

RRRd

11

01

,

:

( ) ( ) pn

i

piip

nnp

yxyxd

RRRd/1

1

0

,

:

∑ =

+

−=

→×


Markus Rupp

Metric Spaces From the lp metrics the l1 and l2 metric as well as

the l∞ metric are of particular interest:

Example 2.5: Hamming Distance

( ) ( )iini

pn

i

piip

nn

yx

yxyxd

RRRd

−=

−=

→×

≤≤

=∞→∞

∞

∑1

/1

1

max

lim,

:

( ) ∑ =⊕=

→×=n

i iiH

nnH

yxyxd

NBBdB

1

0

,

:,1,0


Markus Rupp

Metric Spaces Definition 2.2: A Metric Space (X,d) consists of

a set X and a metric d, valid on this set.

Example 2.6: The set Rn (vectors with n entries from R) together with the metric d2(x,y) builds a metric space.


Markus Rupp

Metric Spaces Note, a metric defines the distance to the zero

vector for p>=1:

A metric thus allows for statements about sizes (lengths,areas, volumes, etc for p=2) of objects. Tied to this is the existence of such objects.

Example 2.7: Consider the infinite long series of real or complex valued numbers xi, i=0,1,...,oo, with the property

∑∞

=

∞<0i

pix

( ) ( ) ( ) pn

i

pi

pn

i

pip xxxd

/1

1

/1

100, ∑∑ ==

=−=

( ) ( ) ∞<= xdxd pp 0,


Markus Rupp

Metric Spaces Together with an lp metric, such sequences form a

metric space called lp(0,∞) space (Ger.: Folgenraum).

Consider on the other hand the sequence xi, i= -∞,...,-1, 0,1,...,∞ with the same properties and same metric, we have the lp(-∞,∞) space.

Consider the metric p ∞, for sequences for which: |xi| is bounded for every i, i.e., |xi|<M, we obtain the l∞(0,∞) or l∞(-∞,∞) space, respectively.

nnnnn yxyxd −=∞ sup),(


Markus Rupp

Supremum and Infimum Definition 2.3. (Supremum): Consider the set S in

R,Q,Z,N… with the elements xi. The smallest number z in R, for which we have:

is called supremum (sup) of the set S. It is the least upper bound.

If there is no number in R that is larger than the largest element in S, then we have sup(S)= ∞.

ixz i allfor ;≥


Markus Rupp

Supremum and Infimum Definition 2.4 (Infimum): Consider the set S in R

with the elements xi. The largest number z in R, for which we have:

is called Infimum (inf) of the set S. It is also called the greatest lower bound.

If there is no more number in R smaller than the smallest element in S, then we have inf(S)=- ∞.

ixz i allfor ;≤


Markus Rupp

Supremum and Infimum Example 2.8: Let S=(3,6) be an open set,

then: inf(S)=3, sup(S)=6

Let T=[4,8), then: inf(T)=4, sup(T)=8 Let U=[2,∞), then: inf(U)=2, sup(U)=∞

Why do we not simply select the maximum (minimum) of S?


Markus Rupp

Metric Spaces Metric spaces can also be formed with particular

properties:

Example 2.9: Let lh(0,∞) be a metric space of sequences in which sequences exist whose quadratic sum is finite (finite energy sequence). Thus, xi in lh(0,∞)=lp=2(0,∞) means, that

∞<∑∞

=0

2

iix


Markus Rupp

Metric Spaces Metric spaces can also be defined over

functions rather than a set of numbers (note that functions are in a way sets of numbers).

These are called metric spaces over functions (Ger.: Funktionenraum).


Markus Rupp

Metric Spaces Definition 2.5 (p-metric): Let X be a set of real-

or complex-valued functions, defined on the interval [a,b] with b>a, with p>=1, that have the property:

The metric over the functions x(t),y(t) from X is then:

(Lebesgue Integral)

∞<≤

−= ∫ pdttytxyxd

pb

a

pp 1;)()(),(

/1

∞<≤∞<∫ pdttxb

a

p 1;)(


Markus Rupp

Metric Spaces The space with the metric dp over the functions is

called Lp space.

For p∞ we have for bounded functions (from above and below) :

For more details on Lp, read Ch 2.1.3

btatytxyxd

txbat

≤≤−=

∞<

∞

∈

;)()(sup),(

)(sup ],[


Markus Rupp

t

0 1 2 3 4 5 6 7 8 9 10

x(t)

0

1

2

3

4

t

0 1 2 3 4 5 6 7 8 9 10

x(t)

0

1

2

3

4

ε

ε

−

+

)()(),(

)(

txtxtx

tx

o

mo

o

)(

)()(

2

1

tx

txtx

oεε

εε

>>>><<

∞

∞

),(),(),(),(

2

1

22

12

xxdxxdxxdxxd

o

o

o

o

εε

<<

∞ ),(),(2

mo

mo

xxdxxd

p-Metrics: consider iso-metricἴ iσος=equal


Markus Rupp

p=1

p=2p=4 p=100

p=1/2

p=1/4

inflation

deflation( )

pp

p

xx

xdxx

x

21

2

1

1

+=

=

=

Metric: Problems with Pytagoras?


Markus Rupp

Under which dp metric is the picture correct?

Metrics for 0<p<1 What happens if we select 0<p<1? In this case

does not satisfy the triangle equality (subadditive property)


Markus Rupp

( ) ( ) pn

i

piip yxyxd

/1

1, ∑ =

−=

Metrics for 0<p<1 Example 2.10: select the three

twodimensional points x=(1,0),y=(0,1) and z=(0.5,0.3)

compute


Markus Rupp

96.3

4

38.2

22

121

57.1

22

121

22

121

=

−+

−>

=

−

∑∑

∑

=

=

=

=

=

=

n

i iin

i ii

n

i ii

yzzx

yx

Metrics for 0<p<1 Definition 2.6: For 0<p<1, the

following expression is a metric:

Example 2.10 again:


Markus Rupp

( ) ( )∑ =−=

n

i

piip yxyxd

1,

( )

798.2

2,

54.1

2

121

25.1

2

121

2

121

21

=

−+

−<

=

−=

∑∑

∑=

=

=

=

=

==

n

i iin

i ii

n

i iip

yzzx

yxyxd

d0-Metric Is there also a d0 metric? Answer: yes but the definition

causes some mathematical difficulties as discontinuities occur for


Markus Rupp

( ) ( )∑ =→ −=n

i

piip yxyxd

100 lim,

=

=elsex

x;1

0;00

Sparseness Definition 2.7: In practise this definition

of d0 is often given in the form of

d0(x)=Σxi0

which is truly a counter for sparseness. But note, this is not a metric (norm)

sometimes it is called a pseudonorm


Markus Rupp

Example 2.11Reconsider Sampling For bandlimited signals we found:

Given the interpolation function, what is the optimal sequence gk to minimize

Answer: gk=fk=f(kT)min=030

Univ.-Prof. Dr.-Ing. Markus Rupp

−

−=

∑

∑

∞

−∞=

∞

−∞=

)(sinc),(min

)(sinc)(

2 kTtT

gtfd

kTtT

ftf

kkg

kk

k

π

π

2/12

)(sinc)(min

−−= ∫ ∑

∞

∞−

∞

−∞=

dtkTtT

gtfk

kgk

π

Example 2.11Reconsider Sampling Given a set of interpolation functions pm(t) :

Find the pair (gk,pm(t));m=1,2,..,M that results in a sparse representation of f(t)!

basic concept for modern speech coding (CELP)


Markus Rupp

( )

TktkTtpgtf

pgd

m

k

kk

Mmmkgk

00

,...,2,10

0);()(

|min

0

≤≤−= ∑=

=

that such

CELPCodebook excited linear prediction We can alternatively also select from a set of

excitation sequences g1,k,g2,k,…,gM,k. The interpolator can be constructed in form of a

fixed part with a linear filter (predictor) before:

The task now is to find the following minimum:


Markus Rupp

−= ∑

−

=

)(sinc)(1

0kTt

Tptp

P

kk

π

−+ ∑

=

0

,0

,2 )(),(minmink

kkmDpg kTtpgttfd

lkm

prediction for tD>0

CELPCodebook excited linear prediction

Problem is split into a search over a fixed set (gm,k) and the minimization of a d2-metricLeast Squares (Chapter 3)


Markus Rupp

21

)(

0

2

0

1

0,

21

)(

0

2

0,

0,2

0 0

,

0 0

,

0

,

)(sinc)(minmin

)()(minmin

)(),(minmin

−−−+=

−−+=

−+

∫ ∑ ∑

∫ ∑

∑

+

=

−

=

+

=

=

TkP k

k

P

l lkmDpg

TkP k

kkmDpg

k

kkmDpg

dtkTlTtT

pgttf

dtkTtpgttf

kTtpgttfd

lkm

lkm

lkm

π


Markus Rupp

Metric Spaces lp(0,∞) : Space of causal sequences lh(0,∞) : Space of causal sequences with

finite energy (p=2) lp(-∞,∞): Space of non-causal sequences

(this includes lp(0,∞)) Lp(0,∞): Space of causal functions Lp(-∞,∞): Space of non-causal functions (C[a,b],dp): Space of continuous functions


Markus Rupp

Metric Spaces Example 2.12 Filter design: a linear time-discrete filter with linear

phase H(exp(jΩ)) =a0+a1exp(jΩ)+a1exp(-jΩ)=a0+2a1cos(Ω) is to design, so that it follows a desired amplitude function |Hd(exp(jΩ))|=1 for |Ω|< ΩG (zero else) most optimally.

( ) ( )

( )

[ ] [ ]

)(;)sin(;

)sin(42)1(min

)cos(2021)cos(21

21min

)cos(221min

21min

10

121

20

20

20,

210

210,

2

10,

2

10

10

10

series Fourier cmp.

Rest

ππ

ππ

ππ

π

ππ

π

π

π

GG

GG

aa

aa

jdaa

jjdH

aa

aaaaa

daadaa

daaeH

deHeH

G

G

Ω=

Ω=

Ω−++−−Ω

=

ΩΩ−−+ΩΩ−−=

ΩΩ−−=

Ω−

∫∫

∫

∫

Ω

Ω−

−

Ω

−

ΩΩ


Markus Rupp

Minimization ofquadratic metric d2

ΩG

Example 2.13. Recall Paley Wiener Given an amplitude function Ad(Ω), what is

a valid filter function F(ejΩ), i.e.Ad

2(Ω) =|F(ejΩ)|2

Different to the previous example the full information of the desired Filter is not known, only the amplitude function is given.

We recall that linear phase filters are well described by their amplitude function.


Markus Rupp

Example 2.13. Recall Paley Wiener Linear phase filters have symmetric

impulse responses


Markus Rupp

Example 2.13. Recall Paley Wiener Problem formulation:

Recall that:


Markus Rupp

Example 2.13. Recall Paley Wiener For which we find:

For l=0,1,…,N. From there we can compute the original coefficients fn.

Thus, given Ad, we now know how to compute F.


Markus Rupp


Markus Rupp

Metric Spaces Convergence of sequences

Definition 2.8 (limit): If there exists to every distance δ a number no so that d(xn,y)< δ for each n>no at fixed value y, the sequence xn is said to be convergent to y.

y is called the limit (Ger.: Grenzwert) of xn. All points in an arbitrarily small distance to y are called the

neighborhood (Ger.: Nachbarschaft) of y.

nn

n

xyyx

∞→=→lim


Markus Rupp

Metric Spaces Example 2.14: the following two sequences

diverge:

Definition 2.9: If a sequence xn returns infinitely often to a neighborhood of z, then we call this point z a limit point (Ger.: Häufungspunkt, Grenzpunkt). E.g., the sequence bn takes on the limit points z=0 and z=2.

If limit points exist, then there must be subsequences (partial sequences, Ger.: Teilfolgen) xn that converge.

nn

n

bna

)1(1

2

−+=

=


Markus Rupp

Metric Spaces The largest limit point of a sequence xn is called

limes superior, or

The smallest limit point of a sequence xn is called limes inferior, or

A sequence converges, if:

nn x∞→suplim

nn x∞→inflim

nnnn xx ∞→∞→ = supliminflim


Markus Rupp

Metric Spaces Example 2.15: Consider the sequence

There are two limit points. The subsequence c0,c2,c4,.. takes on the limit 4, while the subsequence c1,c3,c5,.. takes on the limit 0.

0inflim5,4sup:but4suplim

,...3,2,1;)2(12

===

=−++=

∞→

∞→

nn

nnnn

nn

ccc

nn

c


Markus Rupp

Metric Spaces Definition 2.10: A sequence xn in a metric

space (X,d) is called Cauchy-sequence, if there exists for every ε>0 a N(ε)>0 so that d(xn,xm)<ε for every m,n>N.

It is possible to prove that sequences that converge are Cauchy-sequences. The opposite is not always true.


Markus Rupp

Metric Spaces Example 2.16: Let X=C[-1,1] be a set of

continuous functions and fn(t) a sequence of functions, defined by

Consider the metric space (X,d2) with

>≤≤−+

−<=

ntntnnt

nttfn

/11/1/12/12/

/10)(

( )∫−

−=1

1

22 )()(),( dttgtfgfd


Markus Rupp

Metric Spaces

nn /1/1

1

− t

)(tfn

Example 2.16:


Markus Rupp

Metric Spaces We find

With d20 for large m and n, m-n=k<oo. We thus conclude that it is a Cauchy sequence.

Note, however, the function in the limit is:

This function is not continuous and thus not in X=C[-1,1]. The sequence is thus not convergent in X!

<−

>−

=nm

mnm

nmn

nm

tftfd mn

;6

)(

;6

)(

))(),((

3

2

3

2

2

>=<

=0102/100

)(ttt

tf


Markus Rupp

Metric Spaces The reason why the Cauchy-sequence does

not converge in X, can be interpreted as a hole in the set of functions X.

If we had defined X as the set that included discontinuous functions as well, then we had obtained a Cauchy sequence that converges.

Definition 2.11: A metric space (X,d) is called complete (Ger.: vollständig), if every Cauchy-sequence converges in X.


Markus Rupp

Resume Question: Is the interpolation of sampled values a

linear operation? Answer: Yes

)()(

)()(

)()(

)2()1(

)2()1(

)2()1(

)2()1(

tftf

kTtpfTßkTtpfT

kTtpßfTkTtpfT

ßff

kk

kk

kk

kk

kk

βα

α

α

α

+=

−+

−=

−+−

→+

∑∑

∑∑∞

−∞=

∞

−∞=

∞

−∞=

∞

−∞=


Markus Rupp

Resume Question: Is the sampling of a continuous function

a linear operation? Answer: Yes

[ ] )2()1()2()1(

)1()1(

)()()]([

kk

k

ßfftßftfSftfS

+=+

=

αα


Markus Rupp

Resume Question: Is the conversion from fk to f(t)

and vice versa unique under equidistant sampling?

Answer: Yes

What is the answer for equidistant sampling at arbitrary start point f(kT+To)?

)(

)()(

kTff

kTtpfTtf

k

kk

=

−= ∑∞

−∞=


Markus Rupp

Resume Note: Many different sequences fk

(i) ;i=1,2,… can lead to the same time-continuous function f(t). However, always a different interpolator is required.

Example:

)2/();(

)(

)()(

)2()1(

)2()2(

)1()1(

TkTffkTff

kTtpfT

kTtpfTtf

kk

kk

kk

+==

−=

−=

∑

∑∞

−∞=

∞

−∞=

Resume Given f(kT)=fk, how do we get

gk=g(kT)=f(kT+T/2)?


Markus Rupp

−+=

+= ∑

∞

−∞=

mTTkTT

fTkTfgm

mk 2sinc

2π


Markus Rupp

Resume Consider the amplitude function of a filter:

A(Ω)=1+cos(Ω)+2cos(2Ω) Question 1: is this a symmetric filter? Answer: potentially yes, but only if

dφ(Ω)/dΩ =const. Question 2: What is the impulse response

if this is a causal symmetric filter? Answer: h0=1; h1=0,5; h2=1; h3=0,5; h4=1.

or further delayed versions of this


Markus Rupp

Resume Consider symmetric impulse response: h0=a, h1=1, h2=a. Which values of |a|<0,5 deliver a linear phase filter?

h0+h1exp(-jΩ)+h2exp(-2jΩ) =exp(-jΩ)[h0exp(jΩ)+h1+h2exp(-jΩ)]

=exp(-jΩ)[a exp(jΩ)+1+a exp(-jΩ)] =exp(-jΩ) [1+2a cos(Ω)]

=exp(jφ(Ω)) A(Ω) φ(Ω)=-Ω

What happens for |a|>0,5? A(Ω)=|1+2a cos(Ω)|Thus, for each value of a from R we have a

linear phase system but the phase function may jump.


Markus Rupp

Resume Consider symmetric impulse response: h0=a, h1=1, h2=-a. For which values of a is this a linear phase system?

h0+h1exp(-jΩ)+h2exp(-2jΩ) =exp(-jΩ)[h0exp(jΩ)+h1+h2exp(-jΩ)]

=exp(-jΩ)[a exp(jΩ)+1-a exp(-jΩ)] =exp(-jΩ) [1+2aj sin(Ω)]

Thus, for every value of a from C with Real(a)=0, this is a linear phase system. Since we have:

exp(-jΩ) [1+2aj sin(Ω)] =exp(jφ(Ω)) A(Ω) φ(Ω)=-Ω


Markus Rupp

Resume Consider the following sequences in the

metric space l∞(1,∞):

Consider a classical Gaussian noise sequence from N(0,σ2). Is such a sequence in the metric space lp(-∞,∞)?

1),(

1;

11

≥−>

+=

+=

∞ nyxyxd

nny

nx

nnnn

nn

allfor ;that Proof


Markus Rupp

Resume What can be said about the convergence of

these causal sequences (n=1,2,..)?

Which of these sequences is in the metric space l2(1,∞), which in l∞(1,∞)?

[ ]

[ ]

n

n

n

n

nn

nn

nd

nc

nb

na

235,0

11

)1(111000

11

)1(111

+=

+=

−−

++=

−−+=


Markus Rupp

Resume Consider the following sequence in Q in the

metric space (Q,d∞):

? in sequence-Cauchya thisIs

converge? sequence thisDoes

,...100000141421,

1000014142,

10001414,

100141,

1014,

11

Q

Not in Q

Yes!

Resume Problems in Q: Consider Is this convergent in Q? Similarly, consider


Markus Rupp

21

1n

sm

nm ∑

=

=

!1

0 ns

m

nm ∑

=

= en

m

nm =∑

=∞→ !

1lim0

61lim

2

21

π=∑

=∞→ n

m

nm


Markus Rupp

Metric Spaces Gibbs‘ Phenomenon You are probably well aware of the fact that

periodic functions can be described by Fourier series.

Example 2.17: Consider the following periodic signal of period 1 in L[-0,5 , 0,5]:

with the Fourier-series:

<<≤≤−−<≤−

=5,025,0025,025,0125,05,00

)(tt

ttf

( )

−−

−+= ∑

=

+

tkk

tfn

k

k

n ππ

)12(2cos12

)1(25,0)(1

1

Josiah Willard Gibbs (11.2.1839 –28.4.1903) was an American scientistin Physics, Chemistry, Mathematics


Markus Rupp


Markus Rupp

Metric Spaces Let‘s run this to noo Note that in this case we have:

0))(),((lim0))(),((lim 2

>=

∞∞→

∞→

tftfdtftfd

nn

nn

-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5-0.2

0

0.2

0.4

0.6

0.8

1

1.2

)(500 tfn=

Example 2.18 Let us consider the problem of finding a

square root of a number A (e.g. A=3 in N) We thus want to find a solution for x2-A=0 An iterative algorithm (Heron) can be

derived (Newton-style):


Markus Rupp

21n

n

nxAx

x+

=+

Heron from Alexandria (Mechanicus; † > 62) Greek mathematician and engineerMethod is known since 1700 BC.

https://de.wikipedia.org/wiki/62

Example 2.18 Obviously, starting with a value from

Q, results in a new value from Q:

For noo, we obtain a value from R!


Markus Rupp

QabA

ba

xbax

xAx

x

nn

nn

n

∈+

=→=

+=

+

+

2

2

1

1


Markus Rupp

Metric Spaces Robustness (Ger.: Robustheit):

Often it is important not to optimize the mean value (e.g., mean data rate, BER) but the value at worst case condition.

If a systems shows to be insensitive to the worst cases, it is called robust.


Markus Rupp

Metric Spaces Robustness is often measured in terms of energy Let xi be the input sequence and yi the output sequence of a

system T. The output is assumed to be distorted by an unknown sequence vi. The form of distortion is not necessarily additive or even known. Furthermore, the M initial states zi of this system are also not known. Assume that there exists a reference system without distortion having an output signal yi

(R). The influence of such distortion can be described be the

following expression:( )

2

0

2

1

2

0

2)(

γ≤+

−

∑∑

∑

==

=N

ii

M

ii

N

ii

Ri

vz

yy


Markus Rupp

Metric Spaces Or somewhat more precise:

Assume now that there exists several possibilities for the realization of this system T(F) depending on a specific strategy. Then the robustness criterion reads:

( )2

0

2

1

2

0

2)(

),0(),,1(sup γ≤

+

−

∑∑

∑

==

=

∈∈N

ii

M

ii

N

ii

Ri

NlvMlz vz

yy

hh

( )

∑∑

∑

==

=

∈∈ +

−

N

ii

M

ii

N

ii

Ri

NlvMlzFvz

yy

hh

0

2

1

2

0

2)(

),0(),,1(supinf


Markus Rupp

Example 2.19: Robustness A power amplifier PA in a cell phone (Ger.: Handy)

can be described by the following nonlinear mapping:

The variable ρ= ρi(T) describes the influence of temperature, aging and more.

In order to define the robustness of the output signal with respect to ρ, the following expression is considered:

iiii

iii vx

xx

y ++

= 2

2

1 ρ

ρ

( )2

0

2

0

2

0

2)(

),0(,sup γ

ρρ≤

+

−

∑∑

∑

==

=

∈N

ii

N

ii

N

ii

Ri

Nlv v

yy

h

PA yiT


Markus Rupp

Example 2.19: Robustness If it is possible to reduce the influence of

ρ by new technologies or improved circuitry, a smaller factor γ is the result:

The improvement is achieved by higher robustness (smaller γ).

Note: Robustness is not sensitivity.

( )22

0

2

0

2,

0

2)(

),0(,sup γγ

ρρ<≤

+

−

∑∑

∑

==

=

∈newN

ii

N

inewi

N

ii

Ri

Nlv v

yy

h

PA yiT

Contr.

+

Binary operation Definition 2.12: A binary operation *

on a set S is a rule that assigns to each ordered pair (a,b) of elements from S some element from S. Since the operation ends with an element in the same set, we call this also a closed operation.

Example 2.20: S=Z, *: a*b=min(a,b)


Markus Rupp


Markus Rupp

Vector Spaces Definition 2.13: A linear vector space S over a set of

scalars T (C,R,Q,Z,N,B) is a set of (objects) vectors together with a binary additive „+“ and a scalar multiplicative „.“ operation, satisfying the following properties:

1) S is a group under addition. 2)

3) W.r.t. the multiplicative operation there exists an Identity (One) and a Zero element:

yaxayxaxbxaxbaxabxbaSxa

SyxTba

+=++=+=∈

∈

)()(,)()(,

:,, have we in and everyFor

00;1 =•=• xxxIdentity elementw.r.t „+“!


Markus Rupp

Groups Definition 2.14 (Group): A set S for which a binary

operation * (operation w.r.t two elements of S) is defined, is called a group if the following holds: 1) for each a,b in S it holds that: (a*b) in S 2) there exists an identity element e in S, so that for every

element a in S: e*a=a*e=a. 3) for each element a in S there exists an inverse element b in

S, so that: a*b=b*a=e. 4) The binary operation * is associative, i.e.,: (a*b)*c=a*(b*c)

The group is denominated by (S, *). If, furthermore, for each pair a,b in S it holds that a*b=b*a

(commutativity), then the group is called commutative or Abelian (Ger.: Abelsch).


Markus Rupp

Groups Example 2.21: Let S be the set of vectors of a

particular dimension. S forms a group w.r.t the additive operator, if the following properties are satisfied:

( ) ( )zyxzyxSzyx

xyyx

SySxxxx

S

SyxSyx

++=++∈

−==+

∈∈=+=+

∈+∈

:that holdsit eachFor :eassociativ is operation additive The

that so , second a exists there element eachFor

that so , in element identity an exists There

:that holdsit everyFor

,,)4

;0

)300

0)2

,)1


Markus Rupp

Rings Definition 2.15 (Ring): A set S for which two

binary operations + and * are defined, is called a ring if the following holds: (S,+) is a commutative group (Abelian) The operation * is associative. Distributivity holds w.r.t. + for all scalars:

a(b+c)=ab+ac, (a+b)c=ac+bc. A ring is denominated by (S,+, *). Note: for * there does not need to be an identity

or inverse element. If there exists additionally the inverse element to

*, then it is called skew field or division ring (Ger.: Schiefkörper).


Markus Rupp

Fields Definition 2.16 (Field): A set S equipped

with two binary operations + and * is called a field (Ger.: Körper), if: 1) (S,+) is an Abelian group 2) (S\0,*) is an Abelian group 3) The operations + and * distribute.

If the set is finite, i.e., |S|<oo, we talk about finite groups, finite rings, finite fields….


Markus Rupp

Fields Example 2.22: The Galois Field GF(2) is a

field. The elements of the field are 0 and 1. Define the operations + and *

a b a+b0 0 00 1 11 0 11 1 0

a b a*b0 0 00 1 01 0 01 1 1


Markus Rupp

Fields W.r.t. + S needs to be an Abelian Group:

a+b is in S. Identity element is 0: 0+0=0, 1+0=1. Inverse element exists: 0+0=0,1+1=0. Associativity: (a+b)+c=a+(b+c). Check the remaining properties yourself...


Markus Rupp

Fields Example 2.23:

(R,+,*), (Q,+,*),(C,+,*) are fields; (N,+,*),(Z,+,*) not!


Markus Rupp

Group

Relation to each other

Ring AbelianGroup

Vector space

SkewfieldField


Markus Rupp

Vector Spaces Example 2.24: for finite dimensional linear vector

spaces: 1) Consider the linear vector space in R4 (set of

quadruples, Ger.: Menge der Quadrupel)

2) The set of m X n matrices with real-valued elements.

3) The set of polynomials of degree 0..n with real-valued coefficients.

=+

=+

−

=

=

2121913

23;

0476

;

2025

;

2451

yxyxyx


Markus Rupp

Vector Spaces Still Example 2.24: Consider the (3,2)

single parity check code in GF(2)3 with the elements:V=[000],[011],[101],[110]

„+“ now means „exclusive or“

Do the remaining elements W=[111],[100],[010],[001] also form a linear vector space?


Markus Rupp

Vector Spaces Example 2.25: for infinite-dimensional linear

vector spaces:

1) Consider the set of infinitely long sequences

xn

2) The set of continuous functions over the

interval [a,b] C[a,b]

3) The set of functions in LpLp[a,b]


Markus Rupp

Vector Spaces Definition 2.17: Let S be a vector space. If V is a

subset of S such that V itself is a vector space then V is called a subspace (Ger.: Unterraum) of S.

Example 2.26: Let S be the set of polynomials of arbitrary degree (>6) and V the set of polynomials of degree less than 6. Then V is a subspace of S.

Example 2.27: A (n,k) binary linear block code is a k-dimensional subspace of GF(2)n.


Markus Rupp

Vector Spaces Definition 2.18: Let S be a linear vector space

over R and T a subset of S. A point x of S is called linear combination (Ger.: Linearkombination) of points in T, if there is a set of points pi ;i=1,2,...,m in T and a finite set of scalars ci ;i=1,2,...,m in R, so that:

Note that the set T does not need to be finite.

mm pcpcpcx +++= ...2211


Markus Rupp

Vector Spaces Example 2.28: Let S=C(R), the set of continuous

functions over the complex (real) numbers. Let, furthermore, p1(t)=1; p2(t)=t, p3(t)=t2. A linear combination of such functions is given by:

Consider the polynomial x(t)=-1+5t+t2, and the function p4(t)=t2-1.

Obviously, the description is not unique. The number of required coefficients varies.

2321)( tctcctx ++=

)()(5)(2)()(5)(

)()(5)()(

42

4321

321

tptptptptptp

tptptptx

+=+−+=

++−=


Markus Rupp

Vector Spaces Definition 2.19: Let S be a linear vector space

and T a subset of S. The subset T (pi ;i=1,2,…,m) is said to be linearly independent (Ger.: linear unabhängig), if for each nonempty linear subset T the only finite set of scalars satisfying the equation

is the trivial solution c1= c2 =...= cm =0.

If the above equation is satisfied by a set of scalars that are not all equal to zero, then the subset (pi ;i=1,2,...,m) is called linearly dependent(Ger.: linear abhängig).

0...2211 =+++

mm pcpcpc


Markus Rupp

Vector Spaces Example 2.29: The previously presented

polynomials p1(t)=1; p2(t)=t, p3(t)=t2, p4(t)=t2-1 are linearly dependent, since:

The polynomials p1(t),..., p3(t) are linearly independent!

The vectors p1=[2,-3,4],p2=[-1,6,-2] and p3=[1,6,2] are linearly dependent since:

0)()()( 431 =+− tptptp

0354321

=++ ppp


Markus Rupp

Vector Spaces At this point several interesting questions arise:

Under which conditions is the linear combination of vectors unique?

Which is the smallest set of vectors required to describe every vector in S by a linear combination?

If a vector x can be described by a linear combination of pi ;i=1..m, how do we get the linear weights ci ;i=1..m?

Of which form do the vectors pi ;i=1..m need to be in order to reach every point x in S?

If x cannot be described exactly by a linear combination of pi ;i=1..m, how can it be approximated in the best way (smallest error)?


Markus Rupp

Resume Which of the following sets of vectors are LI and which not?

−

12363

,

9642

,

4321

137

,1

5,

53

,21

2000

,

3300

,

4220

,

5111

4321

,

0321

,

0021

,

0001


Markus Rupp

Resume Given the set pi and a vector x, how do we obtain the linear

weights?

[ ]

[ ]

[ ] [ ]( ) [ ] xppppppppp

xppp

xppp

xppp

H

mm

H

m

m

m

m

m

mm

,...,,,...,,,...,,

,...,,

,...,,

...

21

1

2121

1

21

1

1

21

2211

−

−

=

=

=

=+++

α

α

α

α

ααα


Markus Rupp

Resume: LI Consequence of missing LI in linear systems of

equations:Rows < Columns

equations of system thefor solution a be Let 00

3

2

1

4

3

2

1

3431

1411

≠

=

x

bbb

xxxx

aa

aa

( )equations. of system thisfor exists solutions ofamount infinite an Thus,

:that have alsomust we The,

that soexist must a vectors, column the in dependencylinear to due

**

*

.0

.000

0

*00

3431

1411

bbxxAxA

xAxaa

aax

=+=+=

==

≠

α


Markus Rupp

Resume: LI? Static wireless channel with equalizer

( ) ( ) ( )( )( )

=

⋅=

+++=

++=

−

−−

−−−−

010

1

1

0

1

01

0

211

1100100

110

110

11

gg

hhh

hq

qghqghghghqggqhhqHqG

D

D=1 3 Equationsfor2 Variables!!!


Markus Rupp

Resume: LI?( ) ( ) ( ) ( )( )( ) ( )( )( ) ( )

( )

=

=++

+++++=

+++++=

+

−−

−

−−−−

−−−−

010

1

)2(1

)2(0

)1(1

)1(0

)2(1

)1(1

)2(0

)2(1

)1(0

)1(1

)2(0

)1(0

2)2(1

)2(1

)1(1

)1(1

1)2(0

)2(1

)2(1

)2(0

)1(0

)1(1

)1(1

)1(0

)2(0

)2(0

)1(0

)1(0

1)2(1

)2(0

1)2(1

)2(0

1)1(1

)1(0

1)1(1

)1(0

12

12

11

11

gggg

hhhhhh

hh

qqghghqghghghghghgh

qggqhhqggqhhqHqGqHqG

D

3 Equationsfor4 Variables!!!OK

LI? We thus have linearly dependent

columns The potential space for solutions is

therefore infinitely large What happens if we make the

equalizer even larger (longer)?


Markus Rupp


Markus Rupp

If the equalizer is too long?( ) ( ) ( ) ( )

( )( ) ( )( )( ) ( )

( ) ( )

=

=++++++

+++++=

+++++++=

+

−−−

−

−−−−−−

−−−−

0010

1

)2(2

)2(1

)2(0

)1(2

)1(1

)1(0

)2(1

)1(1

)2(0

)2(1

)1(0

)1(1

)2(0

)2(1

)1(0

)1(1

)2(0

)1(0

3)2(2

)2(1

)1(2

)1(1

2)1(2

)1(0

)2(2

)2(0

)2(1

)2(1

)1(1

)1(1

1)2(0

)2(1

)2(1

)2(0

)1(0

)1(1

)1(1

)1(0

)2(0

)2(0

)1(0

)1(0

2)2(2

1)2(1

)2(0

1)2(1

)2(0

2)1(2

1)1(1

)1(0

1)1(1

)1(0

12

12

11

11

gggggg

hhhhhh

hhhhhh

qqghghqghghghghqghghghghghgh

qgqggqhhqgqggqhhqGqHqGqH

D

4 Equationsfor6 Variables!!!OK


Markus Rupp

Vector Spaces Example 2.30: Consider the complex numbers

z=r+j i in R2 in vector form:

This is a set describing z and z*. Note that both elements are linearly independent as long as i is unequal to zero!

−

=

ir

ir

T ,


Markus Rupp

Vector Spaces Example 2.31: Consider band-limited functions or

sequences xk (also band-limited random processes) that only exist in a frequency range S. For a sequence fk (linear time-invariant system) existing in the complementary space of S we find:

Band-limited signals are said to be linearly dependent! Note however, that the property “finite number of

elements” is not necessarily satisfied.

( ) ( )( ) ( ) ( )

knk

kknk

kn

jjjkn

kkn

jjn

ff

ff

eXeFeYf

SeFSeX

−

−

−∞=−

∞

=

ΩΩΩ−

∞

−∞=

ΩΩ

∑∑

∑

−−=

==⇔==

∈Ω=∉Ω=

xxx

xy

for for with xLet

1

010

11

00

0;0


Markus Rupp

Example 2.33 Consider blind channel estimation (for

equalization):

h1

h2

sk

g1

g2

sk(1)

sk(2)

-0

rk(1)

rk(2)


Markus Rupp

Example 2.33 Observe N receive values at each sensor:

rk(1)..=[rk

(1),rk-1(1)…,rk-N+1

(1)]T

We obtain the equation:

Which properties must the receive vectors rk(1) and rk

(2)

have, so that the problem can be solved uniquely? Vectors rk

(1,2) need to be linearly dependent!

[ ] 0,...,,,,...,,

......

2

21

1

11

)2()2(2

)2(1

)1()1(2

)1(1

)2(2

)2(222

)2(121

)1(1

)1(212

)1(111

==

−

−

+++=+++

gR

g

gg

g

rrrrrr

rgrgrgrgrgrg

m

mmm

mmmm


Markus Rupp








Markus Rupp

System Identification Problem: The impulse response hk of a

linear time-invariant system is to estimate based on observations of input and output signals. What conditions does the input skhave to satisfy to ensure a uniqueidentification?

hsk rk


Markus Rupp

System Identification Solution:

=

−

−− 1

0

1

2

1121 r

hh

hhssss

M

MMM


Markus Rupp


=

−

−

+

−

2

1

0

1

2

1

132

121

rr

hh

hh

ssssssss

M

M

MM

MM


Markus Rupp


For a unique identification, the rows (columns) need to be linearly independent!

=

−

−

−

−−+

−

+

−

M

M

M

M

MMMM

M

MM

MM

rr

rr

hh

hh

sssss

ssssssss

1

2

1

0

1

2

1

12221

1

132

121


Markus Rupp

System Identification A signal with such a property is said

to be of persistent excitation (Ger.: hartnäckige Anregung).

Question: is a complex-valued exponential harmonic sk=exp(jΩοk) of persistent excitation?

What about a sinusoid sk=sin(Ωοk)?

Persistent Excitation Consider excitation signal

only system of order 1 (constant) can be identified.

[ ][ ] 1

322

)1(201

,...,,,,...,,,

xeeeeexeeeex

es

ooooo

oooo

o

jMjjjj

Mjjjj

kjk

ΩΩΩΩΩ

−ΩΩΩΩ

Ω

==

=

=

Univ.-Prof. Dr.-Ing. Markus Rupp145

Persistent Excitation Consider

excitation signal:

only system of order 2 can be identified.

( )[ ][ ]

( )

b

j

a

j

jkjjkjok

ba

Mjjjjb

Mjjjja

kjkjok

xj

exj

ex

eeeej

ks

xj

xj

x

eeeexeeeex

eej

ks

oo

oooo

oooo

oooo

oo

22

21))1(sin(

21

21

,...,,,,...,,,

21)sin(

2

1

1

)1(20

)1(20

Ω−Ω

Ω−Ω−ΩΩ+

−Ω−Ω−Ω−Ω−

−ΩΩΩΩ

Ω−Ω

−=

−=+Ω=

−=

=

=

−=Ω=


Persistent Excitation How many signals (frequencies Ωo) of the

form exp(jΩok) are required to identify a time-invariant system of order M?

How many signals (frequencies Ωo) of the form sin(Ωok) are required to identify a time-invariant system of order M?



Markus Rupp

Vector Spaces Definition 2.20: Let T be a set of vectors in a

vector space S over a set of scalars R(C,Q,Z,N,B). The set of vectors V that can be reached by all possible (finite) linear combinations of vectors in T is the span (Ger.: aufgespannte Menge, erzeugte Menge, lineare Hülle) of the vectors. :

)span(TV =

Vector Spaces Note that this is an elegant way of saying something in short

that otherwise would require a lengthy formulation. If T is a set, then span(T) is a set as well!

The number field is typically K=R or K=C. The span is typically defining a vector space!


Markus Rupp

∈∈==

==

∈+==

=

∑=

21

22

2

11

1

,|)span(

,....,2,1|,|)span(

,

TxKxTV

NixTKyxTV

yxT

iiii

N

i

i

αα

βαβα


Markus Rupp

Vector Spaces Definition 2.21: Let S be a vector space, and let

T be a set of vectors from S such that span(T)=S. If the elements in T are linearly independent, then T is said to be a Hamel basis for S.

Example 2.34: The vectors p1=[1,6,5], p2=[-2,4,2], p3=[1,1,0] and p4=[7,5,2] are linearly dependent. Note that T=p1,p2,p3 spans the space R3 and thus is a basis for R3.

Example 2.35: The vectors p1=[1,0,0], p2=[0,1,0] and p3=[0,0,1] are linearly independent and are a basis for R3. This basis is often called natural basis (Ger.: natürliche Basis).

Georg Karl Wilhelm Hamel (12 September 1877–4 October 1954)

http://en.wikipedia.org/wiki/September_12

http://en.wikipedia.org/wiki/1877

http://en.wikipedia.org/wiki/October_4

http://en.wikipedia.org/wiki/1954


Markus Rupp

Vector Spaces Example 2.36: Consider the (3,2) single parity

check code in GF(2)3 with the elements:V=[000],[011],[101],[110].A Hamel basis is given by:

=]011[]101[

G


Markus Rupp

Example 2.37 The following 3x3 matrices are a Hamel basis in R3x3:

100000000

,010000000

,001000000

000100000

,000010000

,000001000

000000100

,000000010

,000000001


Markus Rupp

Example 2.38 Are these 3x3 matrices a Hamel basis?:

Answer: These matrices are not a basis for R3x3. However, the are a basis for the subspace in R3x3,

that has a zero row and column sum!

−−

−−

−

−

−

−

110110000

,101101000

,110000110

,101000101


Markus Rupp

Vector Spaces Definition 2.22: The number of elements in a set

is its cardinality |A| (Ger.: Kardinalität).

Theorem 2.1: If two sets T and U are Hamel bases for the same vector space S, then T and U are of the same cardinality.

Proof: (only for the finite dimensional case). Letbe two bases for S and with at least one coefficient unequal to zero,

,...,,;,...,,2121 nm

qqqUpppT ==

mm pcpcpcq +++= ...22111

4,1,1,1,1 =−−+−−+= AA :QPSK jjjj


Markus Rupp

Vector Spaces say c1, then: Thus

is also a basis for S. Further substitution leads to

( ),...,,

...1

21

2211

1

m

mm

ppq

pcpcqc

p −−−=

,,...,,,

,,...,,,...

,...,,,

,...,,,

1321

1321

321

321

mm

mm

m

m

qqqqq

pqqqq

ppqq

pppq

−

−


Markus Rupp

Vector Spaces Consequently, we must have: Now starting with q1 instead of p1, we

obtain Thus, we must have n=m.

Definition 2.23: Let T be a Hamel basis for S. The cardinality of T is the dimension of S, |T|=dim(S). It equals the number of linearly independent vectors, required to span the space S.

nm ≥

.mn ≥


Markus Rupp

Vector Spaces Note: each vector space has at least one

Hamel basis.

Operations in the vector space are often simpler and of lower complexity in their basis.


Markus Rupp

Example 2.33 revisited Consider blind channel estimator (blind

equalization):

h1

h2

sk

g1

g2

sk(1)

sk(2)

-0

rk(1)

rk(2)


Markus Rupp

Example 2.33 revisited Observe N receive values at each sensor:

rk(1)..=[rk

(1),rk-1(1)…,rk-N+1

(1)]T

When are the vectors linearly dependent?

In order to obtain linear dependency, we must have a sufficient number of vectors r1

(1)..rm(1), r1

(2)..rm(2), so that

2m>N since the vectors rk(1,2) of dimension N can maximally

span RN!

[ ] mN

m

mmm

mmmm

Rg

g

gg

g

rrrrrr

rgrgrgrgrgrg

2

2

21

1

11

)2()2(2

)2(1

)1()1(2

)1(1

)2(2

)2(222

)2(121

)1(1

)1(212

)1(111

R;0R,...,,,,...,,

......

×∈==

−

−

+++=+++


Markus Rupp

Inner Products Definition 2.24: A vector space for which an

inner vector product is defined is said to be a pre- Hilbert space (Ger.: Innerer Vektorproduktraum, Vor-Hilbertraum).

An inner vector product maps two vectors onto one scalar with the following properties:

zyzxzyx

yxyx

xyyx

xxx

,,,)3

,,)2

,,)1

else. 0 and 0for 0,)0*

+=+

=

=

≠>

αα


Markus Rupp

Inner Products Example 2.39: Consider the space of continuous functions

C[a,b] with the two elements x(t) and h(t). Let x(t) be an input signal and h(t) the impulse response of a low pass. We have:

Example 2.40: Consider vectors in C: Matrices can also build inner products:

∫

∫

==

−=

T

T

gxdgx

dThxTy

0

0

,)()(

)()()(

τττ

τττ

BAABH ,)Trace( =

xyyx H=,


Markus Rupp

Inner Products Example 2.41: Consider the expectation of a

random variable:

This is also an inner vector product, however with an additional weighting function.

[ ]

f

dxdyyxfxy

yx,

),(xyE xy

=

= ∫∫

Inner Products Consider two 2-dimensional vectors x and y. Their inner product (projection) is a

measure of non-orthogonality:


Markus Rupp

000 <=> y,xy,xy,xy

xy

xy

x


Markus Rupp

Complete Space Remember:

Definition 2.11: A metric Space (X,d) is called complete (Ger.: vollständig), if every Cauchy-sequence converges.


Markus Rupp

Hilbert- and Banach Spaces Definition 2.25: A complete normed

(metric) vector space is said to be a Banach space. Is there additionally an inner vector product (the norm is an induced norm), the space is said to be a Hilbert space.

Example 2.42: The space of continuous functions (C[a,b],d∞) is a Banach space.

Example 2.43: The space of continuous functions (C[a,b],dp) is for finite p not a Banach space since it is not complete.


Markus Rupp

Hilbert- and Banach Spaces Example 2.44: The space of sequences lp(0,∞) is a

Banach space. For p=2 it is also a Hilbert space. Example 2.45: The space of functions Lp[a,b] is a

Banach space. For p=2 it is also a Hilbert space.This Hilbert space is often denoted as L2(R) for functions and l2(R) for sequences.

In the following part of the lecture we will exclusively consider Hilbert spaces. (if not noticed otherwise)


Markus Rupp


Markus Rupp

Orthogonality Definition 2.26: Vectors of a pre-Hilbert space

are said to be orthogonal or perpendicular (Ger.: normal) if <x,y>=0.

Definition 2.27: Vectors of a pre-Hilbert space are said to be orthonormal if:

1,

1,

0,

=

=

=

yy

xx

yx


Markus Rupp

Orthogonality Example 2.46: The following set of

vectors is orthogonal:

How do we have to modify the set, in order to make the vectors orthonormal?

−−

−−

−

−

1111

,

1111

,

1111

,

1111

Orthogonality Consider a set of orthogonal vectors.

We then have:

Consider a set of orthonormal vectors. We then have:


Markus Rupp

ii

m

ii

m

iii

m

iii pppp ,,

1

2

11∑∑∑

===

= ααα

∑∑∑===

=m

ii

m

iii

m

iii pp

1

2

11, ααα


Markus Rupp

Orthogonality Example 2.47: The following set of functions is

orthonormal in [-π, π]:1,ejt/4, ej2t/4, ej3t/4

The inner product is defined as:

∫

∫

∫

−

−

−

=−=

−=

==

=

π

π

π

π

π

π

π

π

π

0)4/exp(21

)4/2exp()4/exp(21,

:)4/2exp()( and )4/exp()(Let

)()(21, *

dtjt

dtjtjtgf

jttgjttf

dttgtfgf


Markus Rupp

Resume Consider the following function

in C[-∞,∞]:[ ]

[ ]

( )?,C[- in converge it Doesto? converge sequence this does Where

: for sequence functional the Compute

p

n

n

d

xxnn

xf

/n

xxxf

],

2exp

/21)(

1

21exp

21)(

20

2

2022

∞∞

−−=

=

−−=

π

σ

σπσ

( )?,L[- in converge it Does pd],∞∞

x

-50 -40 -30 -20 -10 0 10 20 30 40 50

f n(x

)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

n=2

n=10

Resume We have to compute dp(fn(t),fm(t)) Select p=2 and find

Cauchy sequence


Markus Rupp

( )

( ) 0)(),(limset

221

21)(),(

2

2

=+=

+−+=

+∞→ tftfdknm

mnnmmntftfd

knnn

mn πππ


Markus Rupp

Resume Cardinality of a set of QPSK?

Dimension of a vector space?= cardinality of the Hamel basis

What is the dimension of the vectors that form a Hamel basis for QPSK ?

What is a linear combination? When are vectors linearly independent?

4,1,1,1,1 =−−+−−+= AA :QPSK jjjj


Markus Rupp

Resume

kNk

k

k

k

MNk

M

M

M

M

k

k

k

Nk

K

K

K

M

M

M

K

K

xHyx

xx

x

hhh

hhh

hh

yyy

y

xxx

xxx

hhhhhh

hhhhh

hhh

yy

yyy

=

=

=

−

−

+−−

−

−

−

−

−

−

+−

−

−

−

−

−

−

1

2

2

01

0

1

01

01

1

2

1

1

2

2

1

0

011

011

011

01

01

0

1

2

1

0

...

......

Which column vectors of these matrices are LI?


Markus Rupp

Example 2.33 revisited Consider blind channel estimator (blind

equalization):

h1

h2

sk

g1

g2

sk(1)

sk(2)

-0

rk(1)

rk(2)


Markus Rupp

Example 2.33 revisited Observe N receive values at each sensor:

rk(1)..=[rk

(1),rk-1(1)…,rk-N+1

(1)]T

When are the vectors linearly dependent?

In order to obtain linear dependency, we must have a sufficient number of vectors r1

(1)..rm(1), r1

(2)..rm(2), so that

2m>N since the vectors rk(1,2) of dimension N can maximally

span RN!

[ ] mN

m

mmm

mmmm

Rg

g

gg

g

rrrrrr

rgrgrgrgrgrg

2

2

21

1

11

)2()2(2

)2(1

)1()1(2

)1(1

)2(2

)2(222

)2(121

)1(1

)1(212

)1(111

R;0R,...,,,,...,,

......

×∈==

−

−

+++=+++

Example 2.33 revisited It turns out in fact that the following three

conditions are sufficient for a solution. 2m>N We now understand that we require to have

persistent excitation of the input signal sk. The Bezout condition must be satisfied for the

channels: no common zero! The solution itself we will provide later in the

context of SVD methods.


Markus Rupp


Markus Rupp

Resume The Hilbert space is a linear vector

space with the following three properties: 1) 2) 3)

Complete (Vollständig)MetricNormed (Normiert)There exists an inner product


Markus Rupp








Markus Rupp

Vector Spaces Definition 2.28: A Hamel basis of dimension m is

said to be orthogonal if for all basis vectors T=p1,p2,p3...,pm:

Definition 2.29: A Hamel basis of dimension m is said to be orthonormal if all basis vectors T=p1,p2,p3...,pm:

=≠≠

=jiji

ppji ;0

;0,

ji

ji jiji

pp

−=

=≠

=

δ;1;0

,


Markus Rupp

Orthogonality Let T=p1,p2,...,pn be a set of vectors. How can we

find a set S=q1,q2,...,qm with m smaller or equal to n so that span(S)=span(T) and the vectors in S are orthonormal?

Gram-Schmidt Method: 1) take p1 and construct

2) build:

3) continue:

4) if ei=0, throw away pi+1.

Erhard Schmidt 13.1.1876- 6.12.1959 German Mathematician

2223

22311332

111211221

1111

,/

;,,

,/;,

,/

eeeq

qqpqqppe

eeeqqqppe

pppq

=

−−=

=−=

=

http://de.wikipedia.org/wiki/1876

http://de.wikipedia.org/wiki/6._Dezember



Markus Rupp

Orthogonality Advantages of orthogonal bases:

[ ]

[ ] . of tionmultiplicaleft by Proof

basis lorthonorma an be let

T

T

ppp

pf

pf

pf

ppp

ppfppfppff

pppPcbaf

321

3

2

1

321

332211

321

,,

,

,

,

,,

,,,

,,],,,[

=

++=

==


Markus Rupp

Vector Spaces Definition 2.30: If there are two bases,

that span the same space with the additional property:

then these bases are said to be dual or biorthogonal (biorthonormal for ki,j=1).

,...,,;,...,,2121 mm

qqqUpppT ==

jijijikqp −= δ,,


Markus Rupp

Vector Spaces Example 2.48: let:

These pairs build a dual basis in R2. Consider

then:

[ ] [ ] [ ] [ ] ;1,0;1,1;1,1;0,12121

TTTT qqpp ==−==

Tbaf ],[=

212211

212211

)(,,

)(,,

qbaqaqpfqpff

pbpbapqfpqff

+−+=+=

++=+=


Markus Rupp

Norms Definition 2.31: Let S be a vector space with

elements x. A real-valued function ||x|| is said to be a norm (length) of x, if the following four properties are satisfied:

Note that just as for metrics 0) follows from 1) and 3)! (can you show this property)?

)inequality (triangle )3

Cfor)2

0 ifonly and if 0)1

every for 0)0

yxyx

xx

xx

Sxx

+≤+

∈=

==

∈≥

ααα


Markus Rupp

Norms Note, norms and metrics are related

concepts:

Example 2.49:

pppp xxdyxyxd =−= )0,(;),(

ii

n

ii

n

ii

xxl

xxl

xxl

max1

2

22

111

=−

=−

=−

∞∞

=

=

∑

∑

:norm

:norm

:norm

[ ] )(sup)(

)()(

)()(

,

2

22

11

txtxL

dttxtxL

dttxtxL

bat

b

a

b

a

∈∞∞ =−

=−

=−

∫

∫

:norm

:norm

:norm


Markus Rupp

Norms Definition 2.32: A normed, linear space is a

pair (S,||.||) in which S is a vector space and ||.|| is a norm on S.

Metrics of a normed linear space are defined by norms.

Definition 2.33: A vector is said to be normalized (Ger.: normiert, unit vector, Ger.: Einheitsvektor) if ||x||=1.

Except of the zero vector all vectors can be normalized.


Markus Rupp

Norms Note: The elements (vectors) of a linear

and normed space are not necessarily normalized vectors.

Only the space is normed, meaning that a norm exists for this space.

Norms Consider a sequence of vectors yk and a

limit y*. To evaluate whether yk converges to the

limit, we employ a norm:

Or, equivalently we can substitute xk=yk-y* and obtain:


Markus Rupp

0lim

0lim *

=

=−

∞→

∞→

kk

kk

x

yy


Markus Rupp

Norms Norm Equivalence Theorem 2.2: Let ||.|| and

||.||‘ be two norms on finite dimensional spaces Rn

(Cn,Qn,Zn,Nn,Bn), then we have:

Proof: It is to show that Without restricting generality we set (Ger.: oBdA)

||.||‘= ||.||2 and obtain for the lower bound:

0'lim,0lim == ∞→∞→ kkkk xx if only and if

2

21

21 1

1

1max

xx

xexexexx

exx

n

ii

n

i

n

iiiii

n

iii

≤

=≤≤≤

=

∑∑ ∑

∑

== =

=

α

α

:Thus

0,;' >≤≤ βαβα xxx

Norms Why is l2 never smaller than l00?

With this method you can prove that


Markus Rupp

∞≥

∞≥+=== ∑∑ xx

xxxxx

i

i

ii

12max

2

max2

2....1

∞≥ xx

p


Markus Rupp

Norms For the upper bound β consider the case:

And we obtain:

With the above condition, x cannot be the zero vector and since ||.||>0 there must be a positive lower bound β:

Since the property is true for the l2 norm, it is also true for all other norms!

02

>= cx

0;122

2

>≥= ββ

xxxxx

xxc β≤=2

'1'1' xxxxxxαβ

βα ≤≤→≤≤


Markus Rupp

Norms Note that the equivalence theorem is defined in terms of

Cauchy sequences. This is a consequence from the proof. Since if a particular norm tends to zero, then it does for every norm. The equivalence is to be understood in this sense.

Norms often find their application in terms of energy relations. It can be in form of average energy (l2-norm) or peak values (l∞-norm).

Thus, norms appear often when we describe systems. (Robustness, nonlinear systems, convergence of learning systems such as adaptive equalizers or controllers).


Markus Rupp

Norms Example 2.50: Hands-free telephone,

Ger.: Freisprechtelefon

+Far end speaker sk rk

Local speaker vk

h -

h

2

2hh − :energyerror System


Markus Rupp

Norms Example 2.51: Consider an adaptive equalizer as

part of a receiver.

The equalizer has a cost function of the form

which it tries to minimize (adaptive process). The error at its output is given by an error vector:

h f+sk rkvk kk ss ≈ˆ

( )kk ssf −ˆ

],...,[]ˆ,...,ˆ,ˆ[ 11 kNkkkNkNkNkNkk eesssssse −+−+−−− =−−−=


Markus Rupp

Norms Example 2.51: Let the adaptive equalizer have

the property that the error signal from k-1 to k is mapped via the learning rule: y=g(x)=x3. Under which condition is the equalizer adaptive?

( )

1......sup

...

...supsup?

21

21

61

61

21

21

22

2

21

2

2

31

31

1

<++++

=++++

=

==

=

−−−

−−−

−−−

−

−∈

−

−−

−

−

Nkk

Nkk

Nkk

Nkk

k

kle

k

Nk

k

k

Nk

k

eeee

eeee

ee

e

eeg

e

ee

hk

For |ek|<1 the error term at the output is always smallerthan at the input.


Markus Rupp

Norms Example 2.52: Consider the linear system in matrix-

vector-form.

Consider the ratio of input and output energy. When do we have a passive system, when an amplifier?

kMNNkN

k

k

k

MNk

M

M

M

M

k

k

k

Nk

xHyx

xx

x

hhh

hhh

hh

yyy

y

,1,

1

2

2

01

0

1

01

01

1

2

1

...

......

−+

−

−

+−−

−

−

−

−

−

−

+−

=

=

)(sup?

2

2,1

2

2,N

kMN

kNlx Hf

x

yhk

=−+

∈

Norms on Matrices In principle we can consider matrices

as vectors and apply the same norms, e.g., apply the l2 norm:

and obtain the Frobenius norm.


Markus Rupp

),min(

...)trace( 222

21

1 1

22

nmp

AAAA pH

m

i

n

jijF

=

++=== ∑∑= =

σσσ

Ferdinand Georg Frobenius (1849-1917)

http://de.wikipedia.org/wiki/Ferdinand_Georg_Frobenius

Norms on Matrices Similarly

define norms on matrices


Markus Rupp

ijij

m

i

n

jij

AA

AA

max'

1 1

'

1

=

=

∞

= =∑∑


Markus Rupp

Induced Norms Inner vector products are helpful for defining

norms. For example, the (quadratic) l2 norm can be written as inner vector product:

This is thus said to be an induced (Ger.: induzierte) norm. The norm is induced by the inner vector product.

Of the lp norms for vectors only the l2 norm is an induced norm!

xxx

xxxxxx n

,

,...

2

222

21

2

2

=

=+++=


Markus Rupp

Induced Norms Example 2.53: induced norm in L2[a,b]

Note that for real-valued signals we have:

How can this be formulated for complex-valued signals?

yxyxyx ,22

2

2

2

2

2=−−+

21

221

2)()(),()(

== ∫

b

a

dttxtxtxtx


Markus Rupp

Induced Norms Vector norms can also be used to induce norms on

multivariate functions. Consider the following example:

( )2,0

expsup

)exp(sup

),(sup

)exp(),(

22

21

2121],[

2

2121],[2

2

21],[

212121

21

21

21

≈+

−=

−=

−

=

−=

=

=

=

xx

xxxx

xxxxx

f

lx

xxff

xxxxxxf

xxx

xxx

xxx

normSelect


Markus Rupp

Induced Norms

-5-4

-3-2

-10

12

34

5

-5-4

-3-2

-10

12

34

50

0.05

0.1

0.15

0.2

0.25

0.3

0.35


Markus Rupp

Induced Vector Norms Matrix-norms are typically induced

vector-norms. Consider matrix A

Note that (spectral norm):

This is the largest singular value.

px

ppx

p

pxpp

xAxxA

x

xAAA

p 100ind,supsupsup =≠≠ =

===

( )AAAxAA Hx maxmax21ind,2

)(sup2

λσ === =

nmR ×∈

Induced Vector Norms


Markus Rupp


Markus Rupp

Spectral Norm Consider firstly a vector with M elements, based on the

sequence exp(jΩk):

In other words, all these vectors are linearly dependent , independent of their dimension M and frequency Ω.

If, however, M different frequencies are selected, M linearly independent vectors can be obtained.

[ ][ ]

[ ] kMT

mkM

kMT

kM

TkM

xmjmMkjmkjmkjx

xjMkjkjkjx

Mkjkjkjx

,,

,1,

,

)exp())1(exp()),...,1(exp()),(exp(

...)exp())2(exp(),...,exp()),1(exp(

))1(exp()),...,1(exp(),exp(

Ω=++−Ω+−Ω+Ω=

Ω=+−ΩΩ+Ω=

+−Ω−ΩΩ=

+

+


Markus Rupp

Spectral Norm Consider now the spectral norm for the vector xM,k:

,ind22,

2,,

2

2,ind2

,

0

1

0

10

,

))(exp(supsup

sup

))(exp(

))(exp())1(exp(

)exp(

))(exp(

))(exp())(exp())1(exp())(exp(

)exp())(exp(

))(exp())1(exp(

...))(exp(0

,AjH

xxA

xxA

A

xjH

nmljmlj

lj

jH

nmljjHmljjH

ljjH

nmlkjmlkj

lkj

hh

hhh

xA

MlkM

lkMMx

Mx

lM

M

M

lkM

kM=Ω→=

=

Ω=

++Ω−+Ω

Ω

Ω=

++ΩΩ−+ΩΩ

ΩΩ

=

+++Ω−++Ω

+Ω

=

Ω∞→+

+

Ω=

=

−

−

+

Further Matrix Norms Recall:

Why is that? Consider an element yi of y=Ax


Markus Rupp

,ind,ind11

,ind

11

1

111

and:BUT

:generalin

max:maxrow

max:maxcolumn

∞∞

=≤≤∞

=≤≤

==

≠

==

=

∑

∑

AAAA

AA

AAA

AA

pp

Tn

jijmi

m

iijnj

?maxmax1

11

==

=→=

∑

∑∑

=

==

m

jjijxix

m

jjiji

m

jjiji

xAy

xAyxAy

jj

We can maximize the expression if sign(xj)=sign(Aij) as then xjAij =|xjAij|

If we further restrict the elements of vector x, e.g., ||x||=1, then there are two possibilities


Markus Rupp

,ind,ind11and

∞∞== AAAA

[ ]

[ ] ijj

jijT

ijj

jijj

T

AxAxx

AxAxx

maxmaxand10,...0,1,0,...,0,0

maxand11,...,1,1

1==→=

==→±±±=

∑

∑∑∞


Markus Rupp

,ind,ind11and

∞∞== AAAA

11

1

1111

1,1

111

11,

maxmaxmax

maxmaxmax

maxmaxmaxmax

maxmaxmaxmax

11

AAAxA

xAxAxxA

A

AAAxA

xAxAxxA

A

m

iijnj

iijj

jjij

ij

jjij

ixxxind

n

jijmi

jiji

jjijxi

jjijixxxind

====

===

====

===

∑∑∑∑

∑∑

∑∑∑

∑

=≤≤

==

∞=

≤≤=

=∞=∞

∞∞

∞

∞∞


Markus Rupp

Resume Norms are…

Metrics in a linear vector space. A norm defined over a linear vector space makes it to a

normed, linear vector space. The knowledge of one norm is in general sufficient due

to the Equivalence Theorem

An induced norm is… a norm, that is build by a „simpler construct“

(example: the inner vector product induces the p=2 norm)

0'lim,0lim == ∞→∞→ kkkk xx if only and if


Markus Rupp

Resume Inner vector products can also be written in terms

of induced norms. In R:

In C:

yxyxyx ,22

2

2

2

2

2=−−+

yxyxyx

yxyxyx

,Re4

,Re22

2

2

2

2

2

2

2

2

2

=−−+

=−−+


Markus Rupp

Induced Vector Norms (Resume) Matrix-norms are typically induced

vector-norms. Consider matrix A

Note that (spectral norm):

This is the largest singular value.

px

ppx

p

pxpp

xAxxA

x

xAAA

p 100ind,supsupsup =≠≠ =

===

( )AAAxAA Hx maxmax21ind,2

)(sup2

λσ === =

nmR ×∈


Markus Rupp

Submultiplicative Property Some of the matrix norms (all p-norms) satisfy

the submultiplicative property:

Note that this property holds for arbitrary matrices, for example

But in this case the p-norms are different functions!

∞=≤ ,...2,1;ind,ind,ind,

pBAABppp

4332 , ×× ∈∈ RBRA

pppxByAxAB ≤

Submultiplicative Property Proof:


Markus Rupp

indpindpp

pxindp

p

pindpx

p

p

p

px

p

pxindp

BAx

xBA

x

yA

x

y

y

yA

xxAB

AB

p

p

pp

,,1,

,1

11,

sup

sup

supsup

==

≤

==

=

=

==


Markus Rupp

Submultiplicative Property Not all matrix norms have this submultiplicative

property: Example 2.54:

∆∆∆

∆

>

==

=

BAAB

BA

AA ij

:have we For 1111

max

Submultiplicative Property A similar property, however, is true

for all unitary invariant norms:

unitary invariant norm:


Markus Rupp

ind

ind

ABAB

BAAB

,2

,2

≤

≤

xxC =


Markus Rupp

Matrix Norms Example 2.55: Consider a causal impulse response hk.

In which case is hk an allpass (amplifier)?

Obviously, the energy of the input sequence must be identical to the energy of the output sequence (for every sequence).

Note that the vectors xN+M-1,k and yN,k have different length!

kMNNkN

k

k

k

MNk

M

M

M

M

k

k

k

Nk

xHyx

xx

x

hhh

hhh

hh

yyy

y

,1,

1

2

2

01

0

1

01

01

1

2

1

...

......

−+

−

−

+−−

−

−

−

−

−

−

+−

=

=


Markus Rupp

Matrix Norms We must thus have for an allpass:

This in turn requires that N∞. For the spectral norm we have at least:

1

supsup

limlimlim

,

,1

,

0,1

,1

0,

,1,1,

==

==

==

−+≠

−+

−+

≠

−+∞→−+∞→∞→

indpN

pkMN

pkN

x

pkMN

pkMNN

xindpN

pkMNNpkMNNNpkNN

H

x

y

x

xHH

xxHy

least at havemust we

have we Since

( ) ( )indNN

jj HeHeH,2

limmax ∞→Ω

Ω∞

Ω ==


Markus Rupp

Matrix Norms An allpass is thus difficult to describe by

matrices of finite dimension or by the spectral norm.

The reason for the finite dimension is that the initial values need also to be considered to reflect the lossless property of the allpass. This property can be required over the entire observation horizon.


Markus Rupp

Matrix Norms Allpasses are better suited for state space forms. Note:

Example 2.56: yk=xk-2

=

+=

+=

+

+

k

kT

k

k

kkT

k

kkk

xz

dcbA

yz

dxzcy

xbzAz

1

1

[ ]

=

+=

+

=

+

+

k

k

k

k

kkk

kkk

xz

yz

xzy

xzz

00110001000,1

10

0010

1

1


Markus Rupp

Matrix Norms Consider the

instantaneous energy:

∑∑==

+

−+

+++++

−+

+

+=+

++=++

++=++

++=++

++=++

=

N

kk

N

kkN

NNNNNN

kkkkkk

kkkkkk

k

k

k

k

xzyz

xzzyzz

xzzyzzxzzyzz

xzzyzz

xz

yz

1

220

1

221

2221

221

2

21

21

221

22

21

2221

221

2

21

21

20

21

22

21

1

001100010

⇐=+

++1

2

2

2

0

2

2

2

1

N

NN

xz

yz

For all N


Markus Rupp

Matrix Norms Allpass as system with feedback

dcbA

q-1

1+kzkz

kx ky


Markus Rupp

Matrix Norms This more general form also allows for describing allpasses

by linear time-variant systems. Example 2.57: Let b=sqrt(1-a2); 0<a<1

.a-1bfor allpass an also describes

allpass. an describes

2kk =

=

−

=

=

−

=

k

kkk

k

kk

kk

k

k

kk

k

k

k

A

xHxx

abba

yy

A

xHxx

abba

yy

)2(

)1(

)2(

)1(

)2(

)1(

)2(

)1(


Markus Rupp

Matrix Norms Allpass

unitary! be to said are conditions these satisfythat , Matrices vectors arbitrary allfor

:havemust we allpass anfor this tocontrast In

have we systems,invariant -timelinear For

N

kMN

pkMN

pkMNN

pkMN

pkN

pkMN

pkN

x

pkMN

pkMNN

xindpN

Hx

x

xH

x

y

x

y

x

xHH

,1

,1

,1

,1

,

,1

,

0,1

,1

0,

1

supsup

−+

−+

−+

−+

−+≠

−+

−+

≠

==

==


Markus Rupp

Small Gain TheoremGeorge Zames(1934-1997) in 1966

Consider the following system with feedback.

Is it stable?

1

0,5q-1

yNhN

gNzN

xN

-

Pole at-0,5

stable!

http://en.wikipedia.org/wiki/George_Zames


Markus Rupp

Small Gain Theorem Consider the following system with feedback.

Is it stable?

y=1/(1+h2)

0,5q-1

yNhN

gNzN

xN

-


Markus Rupp

Small Gain Theorem The „Small Gain Theorem“ can be interpreted as

an extension from linear systems to nonlinear systems for the stability problem.

In linear time-invariant systems the gain of the open loop is responsible for stability (BIBO stability).

The Small Gain Theorem can be applied to non-linear as well as to time-variant systems.


Markus Rupp

Small Gain Theorem Consider for this the input output relation of a

system described by vectors: Definition 2.34: A mapping H is said to be lp-

stable if there exists two positive constants β,γ so that for all input sequences the following is true:

Definition 2.35: The smallest positive value γ, for which lp-stability is obtained, is said to be the gain of H (Ger.: Verstärkung).

Note: BIBO (Bounded Input Bounded Output) stability is equivalent to l∞- stability.

βγ +≤=pNpNNpN

xxHy

NNNxHy =


Markus Rupp

Small Gain Theorem Consider now a system with feedback, comprising

of two systems G and H and corresponding gains γgand γh described by the following signals:

[ ][ ]

NNNNNN

NNNNNN

yuGgGz

zxHhHy

+==

−==


Markus Rupp

Small Gain Theorem

HN

GN +

yNhN

gNzN uN

xN

-


Markus Rupp

Small Gain Theorem Theorem 2.3: (Small Gain Theorem) If both gains

γg and γh are such that:

then the two signals gN and hN of the system with feedback are bounded by:

1<hgγγ

[ ]

[ ]ghhNhNhg

N

hggNgNhg

N

xug

uxh

βγβγγγ

βγβγγγ

+++−

≤

+++−

≤

11

11


Markus Rupp

Small Gain Theorem Proof: Start with:

the norm is given by:

The derivation for gN is analogue.

[ ]

[ ]hggNgNhg

hggNgNNhg

ghNhNgN

gNgN

NNNN

ux

uxh

hux

gx

gGxh

βγβγγγ

βγβγγγ

ββγγ

βγ

+++−

=

++++=

++++≤

++≤

+≤

11

NNNNNN yugzxh +=−= ;

HN

GN +

yNhN

gNzN uN

xN

-


Markus Rupp

Example 2.58 Let the automatic control

of a cell phone power amplifier look like this:

H2

G2 +

y2h2

g2z2 u2=0

x2

-

=

+

+=

37,031

01

5,2

120

2

212

212

221

221

2

G

xx

xx

H

ρρ

ρρ


Markus Rupp

Example 2.58 Is this control stable? We describe G2 and H2 by their gains:

2222

,2

22,2222

222

,2

212

212

221

221

22,2222

37,037,0

31

5,20

15,2

120

gggGz

hh

xx

xx

hHy

ind

ind

ind

ind

=

=≤

=

+

+=≤

ρρ

ρρ


Markus Rupp

Example 2.58

stable.-l thus ,0,372,5:gain loop open the for have thus We

21

37,09137,0

sup37.0

31

5,21

21

5,2

sup

01

5,2

120

sup0

15,2

120

22

21

22

21

2

1

,2

22

21

21

2

212

2122

2

2

221

221

22

2

2

1

212

212

221

221

,2

212

212

221

221

22

21

2

12

<×=

=+

+=

=+

++

+=

+

+

=

+

+

=+

=

γ

ρρ

ρρ

ρρ

ρρ

ρρ

ρρ

xx

xx

xx

xx

xx

xx

h

xx

xx

xx

xx

xx

xx

ind

xx

h

ind


Markus Rupp

Example 2.60 Consider the following problem:

Let the shape of the signal s(t) be known at the receiver.

How is the filter g(t) to design so that the SNR at the output of g(t) is maximum?

matched filter (Ger.: Signalangepasstes Filter)Dwight O. North 1943

g(t)+s(t)

v(t)d(t)

r(t)


Markus Rupp

Chauchy-Schwarz Inequality Theorem 2.4: In an inner-product space S in Cn

with induced norm ||.|| we have:

The inequality becomes an equality if and only if x=αy.

Proof: We start with the simple fact that

22, yxyx ≤

02

≥− yx α


Markus Rupp

Chauchy-Schwarz Inequality We find further that

The minimum is obtained for: Thus:

−

−+−=

+−=−≤

2

2

*

*2

2

2

22

2

2

2

2

2

2

22

2

2

2

,,,

,Re20

y

yx

y

yxy

y

yxx

yyxxyx

αα

ααα

2

2

,

y

yx=α

2

2

2

2

2

2

2

,min0

y

yxxyx −=−≤ αα

22, yxyx ≤→


Markus Rupp

Chauchy-Schwarz Inequality We are not done!

For x=αy we obtain ||x-αy||=0. Due to the norm property , equality to zero

can only be obtained for || x-αy||=0 if the argument is zero.

Thus equality is obtained if and only if x=αy.

0≥x


Markus Rupp

Chauchy-Schwarz Inequality The Cauchy-Schwarz inequality implies:

for S in Cn:

for the space of real-valued functions in the interval [a,b]:

( )( )yyxxyx HHH ≤2

∫∫∫ ≤

b

a

b

a

b

a

dttgdttfdttgtf )()()()( 22

2


Markus Rupp

Chauchy-Schwarz Inequality Example 2.59 Correlation coefficient

Remember: For the correlation coefficient we have:

Proof: Consider zero mean random variablesx‘=x-mx and y‘=y-my:

( )( )[ ]1

E111 ≤≤−⇒≤≤−

yx

yxxy

-y-xσσ

mmr

( )( )[ ]yx

yxxy

-y-xσσ

mmr

E=

[ ] ( )( )[ ]

[ ] [ ][ ] [ ]

'

EEEE

EE

xy22yx

yx

yx

yx

yxxy

y'x'y'x'y'x'

yxxy

r

mmmmr

===

−−=

−=

σσ

σσσσ


Markus Rupp

Chauchy-Schwarz Inequality Chauchy-Schwarz requires that

[ ] [ ] [ ][ ] [ ] [ ]

[ ][ ] [ ] 1

y'Ex'Ey'x'E

y'Ex'Ey'x'E

y'Ex'Ey'x'E

''','

22

22y'x'

222

22

22

≤=

≤

≤

≤

r

yxyx


Markus Rupp

Still Example 2.59 Correlation Coefficient Note that E[.] defines an ensemble average. In practical problems this is often replaced by a

time average (ergodic processes). Cauchy Schwarz still holds then:

( )( )

( ) ( )∑∑

∑

==

=

==

=

N

k

N

k

N

k

mN

mN

mmNr

1

22

1

22

1

1;1

1

ykyxkx

yx

ykxk

xy

-y-x

-y-x

σσ

σσ


Markus Rupp

Back to Example 2.60 Matched Filter (Ger.: Signalangepasstes Filter):

g(t)+s(t)

v(t)d(t)

r(t)

∫∫∫∫

∫

∫

∞

∞−

∞

∞−

∞

∞−

∞

∞−

∞

∞−

∞

∞−

=−≤

−==

−=

ττττττττ

τττ

τττ

dgdrdtgdr

dtgrttd

dtgrtd

)()()()(

)()()(

)()()(

2222

2

02


Markus Rupp

Example 2.60 Matched filter:

g(t)+s(t)

v(t)d(t)

r(t)

)()()()()(

""

)()()( 222

ταταττατ

ττττ

−≈−=⇔=−

=→

≤ ∫∫∞

∞−

∞

∞−

srgrg

dgdrtd

:if only and if Identity,tpoint time suitable a to energyreceiver Maximize 0


Markus Rupp

Example 2.60 Graphical:

r(τ)g(t-τ)

∫∞

∞−

−= τττ dtgrtd )()()(

d(t)

g(t-τ)

τ

t


Markus Rupp

Alternative Example 2.60

g(t)+as(t)

v(t)d(t)

r(t)[ ] [ ]

)(),()(),(')'(

')'()'()()()(

)(),()()()()(

).()()(;1

2222

2

2

2

22

22

ττσττσττσ

τττττδτστττ

ττττττττ

τδστ

ggtgtgdtg

dtgdtgdtg

stgdstgdstg

tt(t) s(t)

vvv

v

v

vEN

:zero is signal if energy Receiver

aES

:zero is noise if energy ReceivervvEaE

with vanda signals random Consider

=−−=−=

−−−=

−=

−=

−=

−=

=+=

∫

∫ ∫∫

∫∫


Markus Rupp

Alternative Example 2.60

g(t)+As(t)

v(t)d(t)

r(t)

2v

2v

)()(2v

2

1)(),(

2v

20

1)(),()(),()(),(

maxmax

:SNR Maximize

)(),(:only noise todueenergy Receiver

)(),()(

:only signal todueenergy Receiver

σσττ

ττσττ

ττσ

ττ

ταττ ==−

=

=

−==

=−=

ssggstg

NS

ggN

stgttS

stgstg


Markus Rupp

Example 2.61 A fixed symbol sequence sk in C with L symbols is

transmitted for the purpose of synchronization at the beginning of a TDMA frame.

Design an optimal synchronization filter fk. Received sequence without noise:

s Infodata ik s …[ ],...i,,i, 21 ssrT =


Markus Rupp

Example 2.61 Correlation filter:

Due to Chauchy-Schwarz we have:

kkk

kkkkkkf

L

lllkk

srf

ffrrfr

frd

k

αα ≈=

=

= ∑−

=+

:thus

,,,max2

1

0

*

Example 2.62Werner Karl Heisenberg (5.12.1901 – 1.2.1976)Hermann Klaus Hugo Weyl, (9.11.1885 – 8.12.1955)

Time-frequency uncertainty:

for F(jω) being the Fourier transform of f(t).


Markus Rupp

21

)(21

)(21

)(

)(

2

22

2

22

≥

∫

∫

∫

∫∞

∞−

∞

∞−∞

∞−

∞

∞−

ωωπ

ωωωπ

djF

djF

dttf

dtttf

21

)(

)(

)(

)(

2

22

2

22

≥

∫

∫

∫

∫∞

∞−

∞

∞−∞

∞−

∞

∞−

ωω

ωωω

djF

djF

dttf

dtttf

Example 2.62 Recall the following properties:

1)

2)

3)

(partial integration with Dirichlet condition that f(t)0 for t+-oo)


Markus Rupp

∫∫∞

∞−

∞

∞−

=−

=−

=

dttfdttfttf

djFdtfjt

jFjdt

tfd

n

nn

nn

n

F

F

)()()(2

)()()(

)()()(

2'

ωω

ωω

Peter Gustav Lejeune Dirichlet13. 2.1805 in Düren; † 5.5.1859 in Göttingen)


http://de.wikipedia.org/wiki/D%C3%BCren


http://de.wikipedia.org/wiki/G%C3%B6ttingen

Example 2.62 Cauchy Schwarz says


Markus Rupp

∫∫∫

∫∫∫∞

∞−

∞

∞−

∫−=

∞

∞−

∞

∞−

∞

∞−

∞

∞−

≤

≤

∞

∞−

dttfdttftdttfttf

dttgdttfdttgtf

dttf

)(')()(')(

)()()()(

222

)(21

2

22

2

22

Example 2.62


Markus Rupp

∫

∫

∫=

∫∫≤

≤

∫−

∞

∞−

∞

∞−∞

∞−

∞

∞−

∞

∞−

∞

∞−∞

∞−

∞

∞−

∞

∞−

∞

∞−

∞

∞−

∫

∫∫

∫∫

ωωπ

ωωωπ

djF

djF

dttf

dttft

dttf

dttf

dttf

dttft

dttfdttftdttf

2

22

2

22

2

2

2

22

2222

2

)(21

)(21

)(

)(

)(

)('

)(

)(

41

)(')()(21

Plancherel

Michel Plancherel (* 16.1.1885; † 4.3.1967 in Zürich) was a Suisse mathematician.

http://de.wikipedia.org/wiki/16._Januar



http://de.wikipedia.org/wiki/Z%C3%BCrich

Example 2.62 Time-frequency uncertainty:

Q: for which functions do we obtain equality?

A: tf(t)=f‘(t)=exp(-αt2)


Markus Rupp

21

)(

)(

)(

)(

2

22

2

22

≥

∫

∫

∫

∫∞

∞−

∞

∞−∞

∞−

∞

∞−

ωω

ωωω

djF

djF

dttf

dtttf

Alternative form (M.Vetterli et al. 2013) Define 1st and 2nd order moment:


Markus Rupp

Also works in the sequence domain:


Markus Rupp


Markus Rupp

nt.tuwien.ac.at - Signal Processing 1 · 2019. 1. 23. · 2 Univ.-Prof. Dr.-Ing. Markus Rupp Learning goals Vector Spaces and Applications of Linear Algebra in Signal Processing (6Units,

Documents