Statistical Signal Processing - 大阪工業大学 ...kamejima/lectureNotes/SSslide.pdf · Statistical Signal Processing An Experimental Mathematics Approach∗ Kohji Kamejima†

Statistical Signal Processing

An Experimental Mathematics Approach∗

Kohji Kamejima†

December 22, 2009

∗†Faculty of Information Science and Technology, Osaka Institute of Technology

2

Linux

3

1 Random Numbers 1

1November 17

4

GPS User Segment: Personal Navigator

5

GPS User Segment: Statistical Analysis

6

SRV Rendezvous

IMGraph.c[img]

7

SRV Network Integration

Vehicle Segment

Relay Site

pseudo range

longitude-latitude

Roadway Segment

Satellite Image

8

SRV Tracking Signal

GGLLT.c[ggllt]

9

GPS: Fluctuation

10

GPS: Bias

11

GPS: Model Error

12

Getting Software Package

First of all create a folder entitled by ISS under properly chosen main folder, say Cprograms.

13


Visit http://www.is.oit.ac.jp/~kamejima/ to download ISS.tar UNIX archive file into ISS.

Extract all files into ISS folder.

All the utility functions are stored in initXWindow.h. Please download it into Cprogramsfolder so that the initXWindow.h file should be included via #include ‘‘../initXWindow.h’’command.

Use PR.c program for the simulation associated with probability and statistics.

Use AR.c program for simulating autoregression model.

The KF.c program can be utilized as the basis for designing filter dymanics.

In the GPS package you can find C programs and toolkit for analyzing GPS signals and relatedionosphere processes.

14

Random Numbers: rand()

Random Number Generator

rt+1 ← (k · rt + c) mod m, (1a)r0 ∈ Z, (1b)

k = 1103515245,

c = 12345,

m = 32768.

mod : modulo operator

15

Coin Game: HT.c

Mathematician Trials FrequencyBuffon 4040 0.507

de Morgan 4092 0.5005Jevons, S. (Economist) 20480 0.5068

Romanovsky 80640 0.4923Pilson 24000 0.5009Feller 10000 0.4947

Program: PRrandomB(double probability )

16

Coin Game (HT.c)#include "./initPR.h"int main(int argc, char *argv[]){....................while(True){....................// Coin Game ..... startstrcpy(text, "T");if(rand()%2==0){strcpy(text, "H");sumH++;

} else {strcpy(text, "T");sumT++;

}// Coin Game ..... end....................}// end of time loopprintf(" End of HT. ButtonPress for Exit, please\n");

}

17

Coin Game (CoinGame.c)CoinGame(int length, int mask, int* ptrEvent, double* ptrValue){....................int Hgain=1;int Tgain=-1;int Event=0;int Gain=0;for(i=0; i<length; i++){....................Rnumber = rand()%2;....................Event = 2*Event + Rnumber;if(Rnumber>0) Gain += Hgain;if(Rnumber<=0) Gain += Tgain;

}....................return (mask & Event)==mask;

}

x(H) Hgainx(T) Tgain∑t

xt Gain

H-T pattern Eventcondition (H***,HH**,. . . ) mask

18

Accident: PR.c, PRdistribution.c

• HH, HHH, HHHH,. . .

Year Total Letters No Address Frequency(×10−6)

1906 983,000,000 26,112 26.561907 1,076,000,000 26,977 25.071908 1,214,000,000 33,515 27.611909 1,357,000,000 33,643 24.701910 1,507,000,000 40,101 26.61

Program: PRrandomBS(int N, double probability )

19

Buffon’s Needles: BF.c, Buffon.c

P =2`

πa (` ≤ a)

Program: Buffon, BuffonNeedle

20

Buffon’s Needles: BF.c, Buffon.c

a

x

0

θ

`

y =x

cos θ

y

P (0 ≤ y ≤ `) =∫ π/2

0

P (0 ≤ x ≤ a | θ)(

dθ

π/2

)=

∫ π/2

0

(min { ` cos θ, a }

a

)(dθ

π/2

)

=

2`

πa

∫ π/2

0

cos θdθ; for ` ≤ a,

???; otherwise.

Program: Buffon, BuffonNeedle

21

Buffon’s Needle (BF.c)#define BUFFON_A 100#define BUFFON_L 50....................// Time LoopSamples = 0; Matches = 0;while(WaitGoExit(&BFW)){Samples++;Condition = BuffonNeedle(BFW, BUFFON_A, BUFFON_L);if(Condition) Matches++;// Statistical Momentssprintf(text, "%4.1f%(%d/%d)",100*(double)Matches/Samples, Matches, Samples);BoxedString(BFW, 0, 10, text, BLUE, WHITE);//getchar();....................}// end of loop....................

22

Buffon’s Needle(Buffon.c)Buffon(double* ptrPoint, int Abuffon, int Lbuffon){double Rshift=Abuffon*Random01();double Rtheta=(PI/2)*Random01();double x=Lbuffon*cos(Rtheta)-Rshift;int cross=(x>=0);double point=0;if((Rtheta<(PI/2))&&(cross))point = x/cos(Rtheta);

....................return cross;

}a

x

0

θ

`

y =x

cos θ

y

a Abuffon` Lbuffonx Rshiftθ Rthetay cross

` cos θ − x x

23

Two Approaches

24

Bertrand’s Paradox

25

Bertrand’s Paradox

Model-1: P =12

Model-2: P =13

Model-4: P = 0Model-3: P =

14

26

Gaussian Process

200 400 t

−1

w

1

Program: PRrandomG()

27

Gaussian Distribution: PR.c, PRdistribution.c

Program the following generator of Gaussian random numbers {wt, t = 0, 1, 2, . . . }based on rt.

wt =

{√Gt; for Gt ≥ 0,

−√

Gt; otherwise,(1a)

Gt = yt

[a0 +

a1

yt + a2

], (1b)

a0 = 2.0611786, a1 = −5.7262204, a2 = 11.640595,

yt = − log[(1 + rt)(1 − rt)]. (1c)

Statistics of random numbers can be designed via “histogram equalization.” Sam-ple values at each iteration time (·) are scattered around “mean value (=0)” with“variance (=1)”.

Program: PRInvGauss(double xpm)

28

Gaussian Distribution: PR.c, PRdistribution.c

P (|y| ≤ σ) ' 0.6826P (|y| ≤ 2σ) ' 0.9545P (|y| ≤ 3σ) ' 0.9973

−5 0y

5

0.5

√2πp

1

Program: PRrandomGcLt(int n)

29

Central Limit Theorem

Program the generator of random numbers given by

yn =r1 + r2 + r3 + · · · + rn√

σ · n. (1)

The histogram of finite sum of independent random numbers converges to the Gauss-ian distribution.

30

Random Number (PR.c)#define BUFFON_A 10#define BUFFON_L 4#define CG_LENGTH 16#define CG_MASK 0x0000 //0x0000 0x8000 0xc000 0xe000 0xf000....................int main(int argc, char *argv[]){....................// Time Loop

....................// Random Numbersvalue = 0.05*PRrandomBpmS(1000, 0.5);dp=0.1*dp0;Condition = CoinGame(Clength, Cmask, &Event, &value);dp=0.01*dp0;Condition = Buffon(&value, ...);dp=dp0;// end of Random Numbers

....................}

31

Random Number (RN.c)RNsample(double* ptrValue){int Mlevel=3, Condition=True;double value=*ptrValue;value = RandomGx2(10);value = RandomGcLt(10);value = 1*RandomG()+0;value = 1*RandomU()+0;value = RandomBpm(0.5);//value = RandomT(10*DT/T_MAX);//value = 0.1*RANDOM_TIME;//value = 4*DT*Casino(0.51, BANKRUPTCHY_SUM);//if(fabs(value)>Mlevel) Condition = True;*ptrValue = value;return Condition;

}

Binary Distribution RandomBpm( probability )Uniform Distribution RandomU()

Gaussian Distribution RandomG() (Central Limit Theorem: RandomGclt( n ))χ2 Distribution RandomGx2( n )

32

2 Stochastic Differential Equation – finite differencescheme – 2

2November 24

33

Simulation Physics: Galilei’s World

GL.c/’G’[gl]

34


Create DE folder in Cprograms.

35


Visit http://www.is.oit.ac.jp/~kamejima to download DE.tar UNIX archive file into DE.

Extract all files into DE folder.

Use GL.c program for simulating Galilei-Aristotle World.

Use DE1.c program for the simulation of first order system.

Use DE2.c program for simulating second order system.

The DE.c program can be utilized as mathematical model of dynamicsl systems.

36

Simulation Physics: Aristotle’s World

GL.c/’A’[gl]

37

Simulation Physics (GL.c)#define GRV -9.8#define GA_MODE ’G’ // ’A’ // Galilei -- Aristotle....................int main(int argc, char *argv[]){

....................while(WaitGoExit(&(GLgraph.wi))){Galilei(GLgraph, GA_MODE, g, t, y, &dydt, dt); //Galilei-Aristotle....................

}// end of t-loop....................return True;

}....................

38

Newton’s World

f = ma, (1)f = −mg, g = 9.8m/s2 (2)

Position : y(t), (3)

Velocity : v(t) =dy(t)dt

, (4)

Acceleration : a =dv(t)dt

=d

dt

(dy(t)dt

)=

d2y(t)dt2

. (5)

Differential Equation : −g = a =d2y(t)dt2

. (6)

Finite Difference :dy

dt∼ y(t + dt) − y(t)

dt, (7)

d2y

dt2∼

dy(t + dt)dt

− y(t)dt

dt. (8)

39

Newton’s World – Simulated

Differential Equation : −g = a =d2y(t)dt2

. (1)

Finite Difference :dy

dt∼ y(t + dt) − y(t)

dt, (2)

d2y

dt2∼

dy(t + dt)dt

− y(t)dt

dt, (3)

Forward Difference : y(t + dt) ∼ y(t) +dy

dtdt, (4)

dy(t + dt)dt

∼ y(t)dt

− gdt, (5)

Programming : y ← y + dydt*dt, (6)dydt ← dydt− g*dt, (7)

40

Simulation Physics (initDE.h)

int Galilei(struct XGRAPH_INFO GR, char mode, double g,double t, double y, double* ptrDydt, double dt)

{double d2ydt2=g;double dydt=*ptrDydt;int radius=5;if(mode==’G’){// Galilei-Newtonif(y<0) d2ydt2 = -2*dydt/dt;dydt += d2ydt2*dt;

}if(mode==’A’){// Aristotleif(y>=Y0) dydt = g;if(y<=0) dydt = -g;

}DrawGraphCircle(GR, t, y, radius, GREEN, WHITE);

....................}

41

2.1 First Order System

42

Dynamical Systems: DE1.c

43


αγ

β

isotope [C14]

44

Differential Equation: First Order System

Differential Equation:

dx(t)

dt= f [x(t)], x(0) = x0.

Simple Example:

dx(t)

dt= −ax(t), x(0) = x0.

Solution:

x(t) = x(0) exp [−at] ,( dx

x= −adt

)

45

Differential Equation: Forward Difference


dx(t)

dt= f [x(t)], x(0) = x0.

Integral Representation:

x(t) = x0 +

∫ t

0

f [x(s)]ds.

Forward Difference:

x(dt) = x(0) +

∫ dt

0

f [x(s)]ds ' x(0) + f [x(0)]dt.

x(t + dt) ' x(t) + f [x(t)]dt.

46


47

Differential Equation: DE1.c -- MainLoop

#define Y0 1.............................int main(int argc, char *argv[]){.............................// t-Loopt = T0;y = Y0;while(WaitGoExit(&(DEgraph.wi))){// Input ... startf = 0;// Input ... end.............................// Differential Equation ... start// First Order Equation.............................dydt = -0.2*y;// Differential Equation ... end.............................

}// end of t-loop}

48

Differential Equation: DE1.c -- Plot

#define Y0 1.............................int main(int argc, char *argv[]){.............................// t-Loopt = T0;y = Y0;while(WaitGoExit(&(DEgraph.wi))){.............................DrawGraphPoint(DEgraph, h, y, GREEN);DrawGraphPoint(DEgraph, h, f, BLUE);.............................

}// end of t-loop}

49

Differential Equation: DE1.c -- Update

#define Y0 1.............................int main(int argc, char *argv[]){.............................while(WaitGoExit(&(DEgraph.wi))){.............................// Dynamicsl Systemdydt += f;// Dynamicsl System ... end// Forward Differencey += dydt*dt;.............................

}// end of t-loop}

50


51

Differential Equation: DE1.c -- Excitation

#define Y0 0.............................int main(int argc, char *argv[]){.............................while(WaitGoExit(&(DEgraph.wi))){// Input ... startf = 0;//f = 1*sin(20*t)+0;// Input ... end.............................

}// end of t-loop}

52

2.2 Second Order System

53


54

Differential Equation: Second Order System


d2y

dt2+ 2ζωn

dy

dt+ ω2

ny = 0, y(0) = y0,dy(0)

dt= y0.

(x1, x2): x1 = y, x2 =dy

dt..

dx1

dt= x2,

dx2

dt=

d2y(t)

dt2= −2ζωn

dy

dt− ω2

ny

= −2ζωnx2 − ω2nx1.

Programming:

x1 : y ← y + dydt*dt,

x2 : dydt ← dydt + d2ydt2+dt,

d2ydt2 = −2ζωny*dt− ω2nx*dt.

55

Differential Equation: DE2.c -- MainLoop

#define Y0 1#define DY0 0.............................int main(int argc, char *argv[]){.............................while(WaitGoExit(&(DEgraph.wi))){

.............................// Input ... startf = 0;// Input ... end.............................// Differential Equation ... start// Second Order Equation.............................d2ydt2 = -2*dydt - 2*y;d2ydt2 = -3*dydt -2*y;.............................// Differential Equation ... end.............................}// end of t-loop

.............................}

56

Differential Equation: DE2.c -- Update

#define Y0 1#define DY0 0.............................int main(int argc, char *argv[]){.............................while(WaitGoExit(&(DEgraph.wi))){// Second Order Systemd2ydt2 += f;// Second Order System ... end// Forward Differencedydt += d2ydt2*dt;y += dydt*dt;.............................}// end of t-loop

.............................}

57


58

Differential Equation: DE2.c -- Excitation

#define Y0 0 // 1#define DY0 0.............................int main(int argc, char *argv[]){.............................while(WaitGoExit(&(DEgraph.wi))){

.............................// Input ... startf = 0;//f = 1*sin(1*t)+0;// Input ... end.............................}// end of t-loop

.............................}

59

Differential Equation: Second Order System


d2y

dt2+ 2ζωn

dy

dt+ ω2

ny = 0, y(0) = y0,dy(0)

dt= y0.

Vector Form:

x1 = y, x2 =dy

dt.

d

dt

[x1

x2

]=

dx1

dtdx2

dt

=

[x2

−2ζωnx2 − ω2nx1

]=

[0 1

−ω2n −2ζωn

] [x1

x2

]

First Order System:

dx

dt= Ax, x =

[x1

x2

], A =

[0 1

−ω2n −2ζωn

].

60

Differential Equation: DE.c -- MainLoop

#define X0 0 // 0.5#define Y0 0 // 0.5.............................int main(int argc, char *argv[]){.............................while(WaitGoExit(&(DEgraph.wi))){.............................// Input ... startf = 0;g = 0;// Input ... end// Differential Equation ... start// System.............................dxdt = y;dydt = -2*x -2*y + 3;.............................// Differential Equation ... end.............................

}// end of t-loop.............................}

61

Differential Equation: DE.c -- Plot

#define X0 0 // 0.5#define Y0 0 // 0.5.............................int main(int argc, char *argv[]){.............................// t-Loopt = T0;x = X0;y = Y0;while(WaitGoExit(&(DEgraph.wi))){.............................DrawGraphPoint(DEgraph, h, x, RED);DrawGraphPoint(DEgraph, h, y, GREEN);.............................

}// end of t-loop.............................}

62

Differential Equation: DE.c -- Update

#define X0 0 // 0.5#define Y0 0 // 0.5.............................int main(int argc, char *argv[]){.............................while(WaitGoExit(&(DEgraph.wi))){.............................// Forward Differencex += dxdt*dt + f*dt;y += dydt*dt + g*dt;.............................

}// end of t-loop.............................}

63

2.3 Stochastic Differential Equation

64


65

Stochastic Differential Equation


dx(t)

dt= f [x(t)] + Gn(t), x(0) = 0,

E{n(t) } = 0, E{n2(t) } = 1.

Integral Representation:

x(t) =

∫ t

0

f [x(s)]ds +

∫ t

0

Gn(s)ds.

Forward Difference:

dx ' f [x]dt + Gndt.

66

Stochastic Differential Equation: White Noise

Formal Representation:

dx = f [x]dt + Gndt,

E{n(t) } = 0, E{n2(t) } = 1.

Brownian Motion:

ndt ∼ σdw,

E{ dw } = 0, E{ dw2 } = dt

σ2E{ dw2 } = σ2dt ∼ E{n2dt2 } = dt2,

σ2 ∼ dt.

n ∼ σdw

dt=

√dt

dw

dt.

67

Stochastic Differential Equation: White Noise

MAthematical Representation:

dx = f [x]dt + Gdw,

E{ dw } = 0, E{ dw2 } = dt.

Equivalent Representation:

dw

dt∼ n√

dt.

dx

dt= f [x] +

G√dt

n.

68

Differential Equation: DE1.c

#define Y0 1.............................int main(int argc, char *argv[]){.............................while(WaitGoExit(&(DEgraph.wi))){// Input ... startf = 0;f = 0.1*RandomG()/sqrt(dt)+0;//f = 1*sin(20*t)+0;// Input ... end.............................

}// end of t-loop}

69

Differential Equation: DE2.c

#define Y0 1#define DY0 0.............................int main(int argc, char *argv[]){.............................while(WaitGoExit(&(DEgraph.wi))){

.............................// Input ... startf = 0;f = 1*RandomG()/sqrt(dt)+0;//f = 1*sin(1*t)+0;// Input ... end.............................}// end of t-loop

.............................}

70

3 Linear Stochastic Systems – from random sequence tosignals – 3

3December 1

71

Linear Stochastic System: AR Model

AR.c[issar]

72

Random Variable (RV.c)int main(int argc, char *argv[]).............................// Time Loop

.............................// Random Numbers.............................value = 0.01*PRCasino(0.51, PR_BANKRUPTCHY_SUM);value = 0.01*PRrandomL(0.01);value = PRrandomT(0.0001);value = 0.5*PR_RANDOM_TIME;value = 0.1*PRrandomBS(1000, 0.001);dp=0.02*dp0;value = 0.05*PRrandomBpmS(1000, 0.5);dp=0.3*dp0;.............................// end of Random Numbers

.............................}

Sum of Binary Random Variable PRrandomBpmS( n, probability )Number of Accidents PRrandomBS( n, probability )

Random Arrival PR RANDOM TIMERandom Jump PRrandomT( probability )Life of System PRrandomL( probability )

Life of Bank PRCasino( probability, assets )

73

Stochastic Process: AR.c

#define DT 0.001#include "./initAR.h"................................intmain(int argc, char *argv[]){//FitISSwindows();AR_DISTURBANCE=False;AR_PARAMETER=False;AR_ESTIMATE_B=False, AR_ESTIMATE_T=False, AR_ESTIMATE_M=False;AR_ERROR=False;return ARmodel();

}

74

Stochastic Process

t

x

75

Stochastic Process

Program stochastic dynamical system governed by

xt+1 = axt + wt, (1)

where {wt, t = 0, 1, 2, . . . } are Gaussian random numbers. Current state is deter-mined by current disturbance and past state. This implies that the effect of randomexcitations is fedback as future disturbances. As the result, stochastic system gen-erates complex signal ( ).

Continuous Systems:

dx

dt= αxt + wt. (2)

76

ARmodel: AR.c

ARmodel.c:

while(t<=t max){

disturbance = w(beta);

signal = x(signal, disturbance);

estimate = My a(signal);

t += dt;}

AR.c:

double My a(double x1){return 0;}int main(int argc, char *argv[]){return ARmodel();}

double w(double Bt){return Bt*sqrt(DT)*RandomG();}

double x(double x0, double w){double dxdt = alpha*x0 + w;return (1+alpha*DT)*x0 + w*sqrt(DT);}

initAR.h:

double ab(double x1);

double at(double x1);

77

Stochastic Process: ARmodel.c

double alpha=-0.05;double beta=50;double x(double x0, double w){double dxdt = alpha*x0 + w;return (1+alpha*DT)*x0 + w*sqrt(DT);

}

double w(double Bt){return Bt*sqrt(DT)*RandomG();

}

xt x( xt−dt, w )α alpha (a ∼ 1 + αdt)dt DTwt w( intensity )

Gaussian Random Number RandomG()

78


intARmodel(){double t=0, dt=DT;double signal=2, disturbance=0;double estimate=0, error=0;...........................// Time Loopwhile((t<=t_max)&&(WaitGoExitT(1000, &(Sgraph.wi)))){disturbance = w(beta);if(!AR_DISTURBANCE) disturbance = 0;signal = x(signal, disturbance);if(AR_ESTIMATE_B) estimate = ab(signal);if(AR_ESTIMATE_T) estimate = at(signal);if(AR_ESTIMATE_M) estimate = My_a(signal);error = 1000*(estimate-parameter)/parameter;.........................................// next tt += dt;

}// end of time loop..................................}

79


intARmodel(){...........................// Time Loopwhile((t<=t_max)&&(WaitGoExitT(1000, &(Sgraph.wi)))){.....................................DrawGraphPoint(Sgraph, t, signal, BLUE);if(AR_PARAMETER) DrawGraphPoint(Sgraph, t, parameter, ADRIATIC_SEA);if(AR_DISTURBANCE) DrawGraphPoint(Sgraph, t, disturbance, RED);if((AR_ESTIMATE_B)||(AR_ESTIMATE_T)||(AR_ESTIMATE_M))DrawGraphPoint(Sgraph, t, estimate, GREEN);

if(AR_ERROR) DrawGraphPoint(Sgraph, t, error, WHITE);.....................................

}// end of time loop..................................}

80

4 Parameter Estimation 4

4December 8

81

Parameter Estimation: AR Model

AR.c[issar]

82

Parameter Estimation ProblemFor linear system,

xt+1 = axt + wt,

minimize square sum of equation errors with respect to the parameter a, i.e.,

xt+1 ' axt =⇒ e2t =

t∑s=0

|xs+1 − axs|2 → min . w.r.t. a. (1)

ytxt−1

wt

axt−1

83

Parameter Estimation: Least Squares

Normal Equation:

∂e2t

∂a= 0 =⇒

t∑s=0

xs+1xs − a

t∑s=0

x2s = 0. (1)

ytxt−1

wt

axt−1

84

Parameter Estimation: AR.c

ARmodel.c:

while(t<=t max){

disturbance = w(beta);

signal = x(signal, disturbance);

estimate = My a(signal);

t += dt;}

AR.c:

double My a(double x1){return 0;}int main(int argc, char *argv[]){return ARmodel();}

double w(double Bt){return Bt*sqrt(DT)*RandomG();}

double x(double x0, double w){double dxdt = alpha*x0 + w;return (1+alpha*DT)*x0 + w*sqrt(DT);}

initAR.h:

double ab(double x1);

double at(double x1);

85

Parameter Estimator: AR.c

................................double My_a(double x1){return 0;

}................................intmain(int argc, char *argv[]){................................AR_ESTIMATE_B=False, AR_ESTIMATE_T=False, AR_ESTIMATE_M=False;................................return ARmodel();

}

86

Impulse Response

Stationary Stochastic System:

dx

dt= −ax + wt,

xt = e−atx0 + Φ(t),

Φ(t) =

∫ t

0

e−a(t−s)w(s)ds.

Steady State:

xt → Φ(t), (t → ∞)

Discrete Model: t = n · dt

Φ(t) =

t/dt∑k=0

φkw(t − kdt),

φk = e−akdt =(e−adt

)k.

87

Parameter Estimation: Successive SchemeThe least squares estimate of the linear system

xs+1 = axs + ws, s = 0, 1, 2, . . . , t

is generated by the following dynamical system:

at = at−1 + P (t)xt[xt+1 − at−1xt],

where P (t) is computed successively as follows:

P (t) = [1 + x2t P (t − 1)]

−1P (t − 1).

yt − yt

yt = at−1xt−1at−1

at

88

Recursive Algorithm

t

pax

axp

89

Stochastic Process: AR.c

#define DT 0.001#include "./initAR.h"................................intmain(int argc, char *argv[]){//FitISSwindows();AR_DISTURBANCE=True;AR_PARAMETER=True;AR_ESTIMATE_B=True, AR_ESTIMATE_T=False, AR_ESTIMATE_M=False;AR_ERROR=True;return ARmodel();

}

90


intARmodel(){double t=0, dt=DT;double signal=2, disturbance=0;double estimate=0, error=0;...........................// Time Loopwhile((t<=t_max)&&(WaitGoExitT(1000, &(Sgraph.wi)))){disturbance = w(beta);if(!AR_DISTURBANCE) disturbance = 0;signal = x(signal, disturbance);if(AR_ESTIMATE_B) estimate = ab(signal);if(AR_ESTIMATE_T) estimate = at(signal);if(AR_ESTIMATE_M) estimate = My_a(signal);error = 1000*(estimate-parameter)/parameter;.........................................// next tt += dt;

}// end of time loop..................................}

91


intARmodel(){...........................// Time Loopwhile((t<=t_max)&&(WaitGoExitT(1000, &(Sgraph.wi)))){.....................................DrawGraphPoint(Sgraph, t, signal, BLUE);if(AR_PARAMETER) DrawGraphPoint(Sgraph, t, parameter, ADRIATIC_SEA);if(AR_DISTURBANCE) DrawGraphPoint(Sgraph, t, disturbance, RED);if((AR_ESTIMATE_B)||(AR_ESTIMATE_T)||(AR_ESTIMATE_M))DrawGraphPoint(Sgraph, t, estimate, GREEN);

if(AR_ERROR) DrawGraphPoint(Sgraph, t, error, WHITE);.....................................

}// end of time loop..................................}

92


double alpha=-0.05;double beta=50;double x(double x0, double w){double dxdt = alpha*x0 + w;return (1+alpha*DT)*x0 + w*sqrt(DT);

}

double w(double Bt){return Bt*sqrt(DT)*RandomG();

}

93

Parameter Estimator: AR.c

#define DT 0.001#include "./initAR.h"

double My_a(double x1){return 0;

}

intmain(int argc, char *argv[]){..............................AR_ESTIMATE_B=False, AR_ESTIMATE_T=False, AR_ESTIMATE_M=True;..............................

}

94

Parameter Estimator: initAR.h

double x0=0;double S00=0;double S10=0;double ab(double x1){double a1=0;S00 += x0*x0;S10 += x1*x0;x0 = x1;if(S00>0) a1 = S10/S00;return a1;

}

95

Recursive Algorithm: initAR.h

double p0=100;double a0=0;double at(double x1){double p1=p0/(1+x0*x0*p0);double e1=x1-a0*x0;double a1=a0+p1*x0*e1;x0 = x1;a0 = a1;p0 = p1;return a1;

}

96

Parameter Estimation: Stochastic ApproximationThe linear system

xt+1 = axt + wt,

can be identified via the following scheme:

at = at−1 + Ktxt[xt+1 − xt+1],

xt+1 = at−1xt,

where Kt is properly designed gain.

yt − yt


at

97

5 Kalman Filtering 5

5December 15

98

Kalman Filter

KF.c[isskf]

99

Parameter Estimation: Successive SchemeThe least squares estimate of the linear system

xt+1 = axt + wt,

is generated by the following dynamical system:

at = at−1 + P (t)xt[xt+1 − at−1xt].

where P (t) is update gain.

yt − yt


at

Problem: How to generate the prediction of stochastic process xt?

100

Kalman Filter

t

pxyx

· yt

xt

xp

101

Kalman Filter (Continuous Time Model)

Stochastic System: Lsystem/initKF.h

dxt

dt= axt + wt, (1a)

yt = hxt + vt. (1b)

Filter Dynamics: KFdynamics/initKF.h

dxt

dt= axt + kt[yt − hxt], (2)

Filter Gain: KFgain/initKF.h

kt = pthσ−1vv , (3)

Error Covariance: KFcovariance/initKF.h

dpt

dt= 2apt + σww − h2p2

t σ−1vv . (4)

102

Kalman Filter (Discrete Time Model)

Stochastic System

xt+1 = axt + wt, (1a)

yt+1 = hxt+1 + vt+1. (1b)

Filter Dynamics:

xt+1 = axt + kt+1[yt+1 − hxx+1], (2a)

Filter Gain:

kt+1 = qth[h2qt + σvv]−1

, (2b)

Error Covariance:

qt+1 = a2pt + σww, pt+1 = [1 − kt+1h]qt+1.

103

Kalman Filter (Stationary Filter)

System with Observation Channel:

dxt

dt= axt + wt, (1a)

yt = hxt + vt. (1b)

Filter Dynamics:

dxt

dt= axt + k∞[yt − hxt], (2)

Stationary Gain:

k∞ = p∞hσ−1vv , (const.)(

dpt

dt→

)0 = 2ap∞ + σww − h2p2

∞σ−1vv .

104

MyFilter (a)


dxt

dt= axt + wt, (1a)

yt = hxt + vt. (1b)

Filter Dynamics: MyKFdynamics/KF.c⇐ KFdynamics/initKF.h

dxt





dpt

dt= 2apt + σww − h2p2

t σ−1vv . (4)

105

Kalman Filter: KF.c

KalmanFilter.c:

disturbance = w(Gt);noise = v(Rt);

Lsystem(disturbance, &signal, noise, &data);

Kt=MyKFgain(pt);

estimate = MyKFdynamics(estimate, Kt, data);

pt = KFcovariance(pt);

KF.c:

double MyKFgain(double p0)

double MyKFdynamics(...)

int main(int argc, char *argv[]){return KalmanFilter();}

initKF.h:

double w(double Gt)double v(double Rt)

double KFcovariance(double p0)

double KFgain(double p0)

double KFdynamics(...)

int Lsystem(...)

106

MyFilter (b)


dxt

dt= axt + wt, (1a)

yt = hxt + vt. (1b)

Filter Dynamics: MyKFdynamics/KF.c⇐ KFdynamics/initKF.h

dxt


Filter Gain: MyKFgain/KF.c

kt = k∞. (3)

107

Filter Gain: KF.c

KalmanFilter.c:



Kt=MyKFgain(pt);estimate = MyKFdynamics(estimate, Kt, data);


KF.c:

double MyKFgain(double p0)double MyKFdynamics(...)


initKF.h:





int Lsystem(...)

108

Stochastic System: KF.c

#define DT 0.001#define At0 -0.001#define Gt0 5.0#define Ht0 1.0#define Rt0 10.0..................int main(int argc, char *argv[]){..................//KF_L_SYSTEM=False;KF_LV_SYSTEM=True; // default: True/FalseKF_DISTURBANCE=False; // default: TrueKF_NOISE=False; // default: TrueKF_SIGNAL=True; // default: FalseKF_DATA=True; // default: TrueKF_ESTIMATE=False; // default: True//KF_MY_ESTIMATE=True;// default: Falsereturn KalmanFilter();

}

109

Stochastic System: KalmanFilter.c

int KalmanFilter(){double t=0, dt=DT;....................................// Time Loopwhile((t<=t_max)&&(WaitGoExitT(1000, &(Sgraph.wi)))){TIME = t;disturbance = w(Gt);if(!KF_DISTURBANCE) disturbance=0;noise = v(Rt);if(!KF_NOISE) noise=0;if(KF_L_SYSTEM) Lsystem(disturbance, &signal, noise, &data);if(KF_LV_SYSTEM) LVsystem(disturbance, &signal, noise, &data);................................................// next tt += dt;

}// end of time loop................................................

}

110


int KalmanFilter(){double t=0, dt=DT;....................................// Time Loopwhile((t<=t_max)&&(WaitGoExitT(1000, &(Sgraph.wi)))){................................................Kt = KFgain(pt);if(KF_MY_GAIN) Kt=MyKFgain(pt);if(!KF_GAIN) Kt=0;estimate = KFdynamics(estimate, Kt, data);if(KF_MY_ESTIMATE) estimate=MyKFdynamics(estimate, Kt, data);pt = KFcovariance(pt);................................................// next tt += dt;

}// end of time loop................................................

}

111


int KalmanFilter(){double t=0, dt=DT;....................................// Time Loopwhile((t<=t_max)&&(WaitGoExitT(1000, &(Sgraph.wi)))){................................................if(KF_SIGNAL) DrawGraphPoint(Sgraph, t, signal, WHITE);if(KF_DATA) DrawGraphPoint(Sgraph, t, data, CLEAR_SKY);if(KF_PT) DrawGraphPointP(Sgraph, t, pt*Pscale, RED);if(KF_ESTIMATE) DrawGraphPoint(Sgraph, t, estimate, GREEN);if(KF_MY_ESTIMATE) DrawGraphPoint(Sgraph, t, estimate, EMERALD_GREEN);................................................UpdateISShistogram(&Dhistogram,disturbance, dp, ’p’, WHITE, BLACK);UpdateISShistogram(&Nhistogram, noise, dp, ’p’, WHITE, BLACK);................................................

}// end of time loop................................................

}

112

Linear Stochastic System: initKF.h

////////////////////////////////////// Linear System//intLsystem(double Gwt, double* xt, double Rvt, double* yt){double dxdt = At*(*xt) + Gwt/sqrt(DT);Gt = Gt0;*xt += dxdt*DT;*yt = Ht*(*xt) + Rvt/sqrt(DT);return True;

}//// end of Lsystem////////////////////////////////////

113

Kalman Filter: initKF.h

double w(double Gt){return Gt*sqrt(DT)*RandomG();

}double v(double Rt){return Rt*sqrt(DT)*RandomG();

}double KFcovariance(double p0){double dpdt = 2*At*p0 + Gt*Gt - p0*Ht*(1/(Rt*Rt))*Ht*p0;return p0 + dpdt*DT;

}double KFgain(double p0){return p0*Ht/(Rt*Rt);

}double KFdynamics(double e0, double Kt, double y){double n=y-Ht*e0;double dedt = At*e0 + Kt*n;return e0 + dedt*DT;

}

114

6 Joint Paramter and State Estimation6

6December 15

115

Ionosphere Instability

Thunder Cloud

n+

e− e−

e− e− e−

e− e−e−

e−

300KV

+

-

+ n+ n- + Electrosphere

Solar RadiationSolar Wind

Cosmic Ray

Kennelly-Heaviside Layer

5780K

He++

1.4KW/m2

n+e−

450Km/s

Rn

Earth

+-

116

Whistler Effect

ISD.c[isd]

117

GPS Deviation: Longitude

118

GPS Deviation: Latitude

119

GPS: Bias

120

Linear System Model

Random Constant:

et = c + vt. (1)


dxt

dt= wt, (2a)

yt = hxt + vt. (2b)

121

Kalman Filter (Continuous Time Model)

dxt

dt= kt[yt − hxt], (1)




dpt

dt= σww − h2p2

t σ−1vv . (3)

122

Kalman Filter (Discrete Time Model)

Stochastic System

xt+1 = xt + wt, (1a)

yt+1 = hxt+1 + vt+1. (1b)

Filter Dynamics:

xt+1 = xt + kt+1[yt+1 − hxx+1], (2a)

Filter Gain:

kt+1 = qth[h2qt + σvv]−1

, (2b)

Error Covariance:

qt+1 = pt + σww, pt+1 = [1 − kt+1h]qt+1.

123

Kalman Filter (Stationary Filter)

System with Observation Channel:

dxt

dt= wt, (1a)

yt = hxt + vt. (1b)

Filter Dynamics:

dxt

dt= k∞[yt − hxt], (2)

Stationary Gain:

k∞ = p∞hσ−1vv , (const.)(

dpt

dt→

)0 = σww − h2p2

∞σ−1vv .

124

Kalman Filter: KF.c

KalmanFilter.c:



Kt=MyKFgain(pt);

estimate = MyKFdynamics(estimate, Kt, data);


KF.c:

double MyKFgain(double p0)

double MyKFdynamics(...)


initKF.h:





int Lsystem(...)

125

Parameter Estimation: Deterministic System (1)

System to be identified:

dxt

dt= −axt,

System model:

dxt

dt= −atxt,

Error Process: (et = xt − xt)

det

dt= −(atxt − axt) = −[(at − a + a)xt − axt]

= −a(xt − xt) − (at − a)xt

= −aet − (at − a)xt,

126

Parameter Estimation: Deterministic System (2)

System and model:

dxt

dt= −axt,

dxt

dt= −atxt,

Liapunov Function: V (t, x) = γte2t + (at − a)2

1

2

dV (t, x)

dt= γtet

det

dt+ (at − a)

dat

dt

= −aγte2t + (at − a)

(dat

dt− γtxtet

),

Adaptation Scheme: V (t, x) ↓ 0

dat

dt= −γtxt(xt − xt).

(et = xt − xt

)

127

7 Project7

7December 22

128

Project

1. Let the output of the dynamical system

xt+1 = axt + wt,

be observable as a time series xt, t = 0, 1, 2, . . .. Program and evaluate the recursive algorithmfor updating the estimate at based on the observation xt and its prediction.

2. Program a version of the Kalman filter for the stochastic system

xt+1 = axt + wt,

yt+1 = hxt+1 + vt+1.

Discuss on the optimality of the filtering gain.

129

8 Summary 8

8January 19

130

Ionosphere Instability

Thunder Cloud

n+

e− e−

e− e− e−

e− e−e−

e−

300KV

+

-

+ n+ n- + Electrosphere

Solar RadiationSolar Wind

Cosmic Ray

Kennelly-Heaviside Layer

5780K

He++

1.4KW/m2

n+e−

450Km/s

Rn

Earth

+-

131

GPS Instability

GPS.c[gps]

132

Nonlinear Dynamics: Predator-Prey

ISD.c[isd]

133

GPS Deviation: Longitude

134

GPS Deviation: Latitude

135

Stochastic System Model

Predator-Prey Dynamics:

dn+

dt= −b(1 − k−e−)n+,

de−

dt= a(1 − k+n+)e−,

Stochastic Population Dynamics:

1n+

dn+

dt= −b(1 − k−e−) + θ+,

1e−

de−

dt= a(1 − k+n+) + θ−.

Stochastic Differential Equation:

dn+

n+= −b(1 − k−e−)dt + Gdw,

de−

e−= a(1 − k+n+)dt + Rdv.

136

Estimatable Model

Stochastic Differential Equation:

dn+ = −b(1 − k−e−)n+dt + Gn+dw,

de−

e−= a(1 − k+n+)dt + Rdv.

State Space Representation:

dn+ = −b(1 − k−e−)n+dt + Gtdw,(Gt = Gn+

)(

adt − de−

e−

)= ak+n+dt + Rdv.

zt = n+, At = −b(1 − k−e−), dy = adt − de−

e−, H = ak+.

bk−e− = b + At,

bk−de− = dAt,

dAt

b + At=

bk−de−

bk−e−=

de−

e−= adt − dy

137

Kalman Filtering

State Space Representation:

dz = Atzdt + Gtdw,

dy = Hzdt + Rdv.

Model Generator:

dAt = (b + At)(adt − dy),

Observation Mechanism:

H = ak+,

Innovation Representation:

dz = Atzdt + Ktdν,

dν = dy − Hzdt.

138

Estimatable Model

System Modeldn+ = −b(1 − k−e−)n+dt + Gtdw(

adt − de−

e−

)= ak+ · n+dt + Rdv

?

?

H = ak+

Model GeneratordAt

b + At= adt − dy

Kalman Filter

? ?dz = Atzdt + Kt[dy − Hzdt]

Estimate-6

Gt = Gz

6

Innovation Representation:

dzt = Atztdt + Ktdνt,

dνt = dyt − Hztdt.

dAt = (b + At)(adt − dy),H = ak+.

Population Dynamics:

dzt

zt= Atdt + Ktdνt,

de−

e−= adt − dyt.

139

Nonlinear Dynamics: Predator-Prey

ISD.c[isd]

140

Kalman Filter: KF.c

KalmanFilter.c:


LVsystem(disturbance, &signal, noise, &data);

Kt=MyKFgain(pt);estimate = KFLVdynamics(estimate, Kt, data);


KF.c:

double MyKFgain(double p0)double MyKFdynamics(...)


initKF.h:




double KFLVdynamics(...)

int LVsystem(...)

141

Stochastic Growth Dynamics: KF.c

#define DT 0.001#define At0 -0.001#define Gt0 5.0#define Ht0 1.0#define Rt0 10.0..................int main(int argc, char *argv[]){KF_DISTURBANCE=True;KF_NOISE=True;KF_L_SYSTEM=False, KF_LV_SYSTEM=True;KF_SIGNAL=True;KF_DATA=True;KF_GAIN=True;KF_PT=True;KF_ESTIMATE=True;return KalmanFilter();

}

142

Stochastic Growth Dynamics: KalmanFilter.c

int KalmanFilter(){double t=0, dt=DT;....................................// Time Loopwhile((t<=t_max)&&(WaitGoExitT(1000, &(Sgraph.wi)))){TIME = t;disturbance = w(Gt);if(!KF_DISTURBANCE) disturbance=0;noise = v(Rt);if(!KF_NOISE) noise=0;if(KF_L_SYSTEM) Lsystem(disturbance, &signal, noise, &data);if(KF_LV_SYSTEM) LVsystem(disturbance, &signal, noise, &data);................................................if(KF_L_SYSTEM) estimate = KFdynamics(estimate, Kt, data);if(KF_LV_SYSTEM) estimate = KFLVdynamics(estimate, Kt, data);................................................// next tt += dt;

}// end of time loop................................................

}

143

Stochastic Growth Dynamics: initKF.h

////////////////////////////////////// Lotka-Volterra System//double alpha=0.3;double beta=0.5;double kx=1.5;double ky=1.5;int LVsystem(double Gwt, double *ptrZ, double Rvt, double *ptrY){.......................double dzdt = At*zt + Gwt/sqrt(DT);double dAdt = (alpha-yt)*(beta+At);At += dAdt*DT;Gt = Gt0*zt;Ht = alpha*kx;zt += dzdt*DT;yt = Ht*zt + Rvt;.......................

}

144

Kalman Filtering: KalmanFilter.c

int KalmanFilter(){double t=0, dt=DT;....................................// Time Loopwhile((t<=t_max)&&(WaitGoExitT(1000, &(Sgraph.wi)))){................................................if(KF_SIGNAL) DrawGraphPoint(Sgraph, t, signal, WHITE);if(KF_DATA) DrawGraphPoint(Sgraph, t, data, CLEAR_SKY);if(KF_PT) DrawGraphPointP(Sgraph, t, pt*Pscale, RED);if(KF_ESTIMATE) DrawGraphPoint(Sgraph, t, estimate, GREEN);if(KF_ESTIMATE) DrawGraphPoint(Sgraph, t, kx, YELLOW);................................................UpdateISShistogram(&Dhistogram,disturbance, dp, ’p’, WHITE, BLACK);UpdateISShistogram(&Nhistogram, noise, dp, ’p’, WHITE, BLACK);................................................

}// end of time loop................................................

}

145

Parameter Estimation: Continuous Time System (1)

System to be identified:

dxt

dt= −axt,

System model:

dxt

dt= −atxt,

Error Process: (et = xt − xt)

det

dt= −(atxt − axt) = −[(at − a + a)xt − axt]

= −a(xt − xt) − (at − a)xt

= −aet − (at − a)xt,

146

Parameter Estimation: Continuous Time System (2)

System and model:

dxt

dt= −axt,

dxt

dt= −atxt,

Liapunov Function: V (t, x) = γte2t + (at − a)2

1

2

dV (t, x)

dt= γtet

det

dt+ (at − a)

dat

dt

= −aγte2t + (at − a)

(dat

dt− γtxtet

),

Adaptation Scheme: V (t, x) ↓ 0

dat

dt= −γtxt(xt − xt).

(et = xt − xt

)

147

Joint Parameter and State Estimation

Volterra’s Principle:1

t1 − t0

∫ t1

t0

(1 − k+n+)dtk+=⇒ 0, n+ ∼ e− ∼ clock bias

Proposition 1 By updating interaction parameter k+ following

dk+

dt= γ+zt(1 − k+zt), (1)

with positive gain γ+ the Kalman filter is adapted to observation dyt ∼ τt.

doubleKFLVdynamics(double e0, double Kt, double yt){double et=KFdynamics(e0, Kt, yt);//return et;double gamma=0.03;double dkdt=gamma*et*(1-kx*et);kx += dkdt*DT;return et;

}

Statistical Signal Processing - 大阪工業大学 ...kamejima/lectureNotes/SSslide.pdf · Statistical Signal Processing An Experimental Mathematics Approach∗ Kohji Kamejima†

Documents