ECNG 6700 - Stochastic Processes, Detection and Estimation Random Processes - Part I Sean Rocke September 26 th & October 3 rd , 2013 ECNG 6700 - Stochastic Processes, Detection and Estimation 1 / 35
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 1/35
ECNG 6700 - Stochastic Processes, Detection and
EstimationRandom Processes - Part I
Sean Rocke
September 26th & October 3rd , 2013
ECNG 6700 - Stochastic Processes, Detection and Estimation 1 / 35
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 2/35
Outline
1 Probability & Random Variables
2 Random Vectors
3 Random (Stochastic) Processes
4 Stationarity
5
Power Spectral Densities for Real WSS Processes
6 Conclusion
ECNG 6700 - Stochastic Processes, Detection and Estimation 2 / 35
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 3/35
Probability & Random Variables
Bounds and Approximations
In many applications where we calculate probabilities we canpotentially run into two problems:
1 We do not know the underlying distributions completely . . . all we
have are sample moments such as E [X ], var (X ) and
higher–order moments E [(X −
µ)k ], k > 2.
2 We know the distributions, but integration in closed form is not
possible (e.g., Gaussian pdf)
Question:
How do we solve this?
We use approximation techniques to establish upper and\or lower
bounds on probabilities. . .
ECNG 6700 - Stochastic Processes, Detection and Estimation 3 / 35
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 4/35
Probability & Random Variables
Bounds and Approximations
At–Home Activity:
Read up on the following inequalities\bounds:
Markov inequality
Tchebycheff (Chebyshev) inequality
Chernoff inequality
Strong Law of Large Numbers (SLLN)
Weak Law of Large Numbers (WLLN)
Central Limit Theorem (CLT)
ECNG 6700 - Stochastic Processes, Detection and Estimation 4 / 35
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 5/35
Probability & Random Variables
Bounds and Approximations
Food for thought:
1 When would you use one approximation technique over another?
2 Which gives a tighter bound: Markov, Tchebycheff or Chernoff?
3 What is the difference between the SLLN and WLLN?
4 Express the CLT in your own words.
5 SLLN, WLLN, and CLT assume IID RVs. What if the RVS are
dependent or not identical?
6 Can you recognize when to use the approximation techniques if
given a problem to solve?
7 Can you apply the appropriate technique(s) when you recognize
that one is necesary?
ECNG 6700 - Stochastic Processes, Detection and Estimation 5 / 35
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 6/35
Random Vectors
Linear Algebra Review
At–Home Activity:
Review the following matrix operations:
1 Transpose
2 Matrix sum3 Matrix product
4 Trace of a matrix
5 Norm of a vector
6 Vector inner & outer products7 Block matrices & operations
ECNG 6700 - Stochastic Processes, Detection and Estimation 6 / 35
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 7/35
Random Vectors
Random Vectors & Matrices
Random Vector:A vector whose entries are RVs
Random Matrix:
A matrix whose entries are RVs
Important Parameters:
Expectation of a vector\matrix
Correlation matrix for a random vector
Covariance matrix for a random vectorCross–correlation matrix for two random vectors
Cross–covariance matrix for two random vectors
ECNG 6700 - Stochastic Processes, Detection and Estimation 7 / 35
R d V
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 8/35
Random Vectors
Random Vectors & Matrices
Example 1:
Write out the correlation, covariance, cross–correlation &cross–covariance matrices for the n–dimensional random vectors
X = [X 1, . . . , X n ]T and Y = [Y 1, . . . , Y n ]
T .
ECNG 6700 - Stochastic Processes, Detection and Estimation 8 / 35
R d (St h ti ) P
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 9/35
Random (Stochastic) Processes
Modelling Uncertainty in Random Signals
A random signal has some element of uncertainty, and thus wecan never determine its exact value at any given time.
Can describe signal probabilistically (e.g., in terms of average
properties, or probability that signal exceeds a given value)
Random Process: The probabilistic model used to describe suchrandom signals
Conceptual Definition of Random (Stochastic) Processes:
Mathematical model of an empirical process whose development isgoverned by probability laws . . .
Let’s look at some examples. . .
ECNG 6700 - Stochastic Processes, Detection and Estimation 9 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 10/35
Random (Stochastic) Processes
RP Example: Modelling Temperature Anomalies
ECNG 6700 - Stochastic Processes, Detection and Estimation 10 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 11/35
Random (Stochastic) Processes
RP Example: Modelling Global Precipitation
ECNG 6700 - Stochastic Processes, Detection and Estimation 11 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 12/35
Random (Stochastic) Processes
RP Example: Modelling Stock Prices
ECNG 6700 - Stochastic Processes, Detection and Estimation 12 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 13/35
Random (Stochastic) Processes
RP Example: Modelling Annual Rainfall
ECNG 6700 - Stochastic Processes, Detection and Estimation 13 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 14/35
Random (Stochastic) Processes
RP Example: Modelling Network Traffic
ECNG 6700 - Stochastic Processes, Detection and Estimation 14 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 15/35
( )
RP Example: Modelling Network Traffic
ECNG 6700 - Stochastic Processes, Detection and Estimation 15 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 16/35
So formally, what is a Random Process?
Definition:
Family of RVs, {X (t ), t ∈ T } defined on a given probability space,S , indexed by the parameter, t , where t varies over the index set,
T Function of two arguments, {X (t , ζ ), t ∈ T , ζ ∈ S}
Questions:
1 For each fixed t = t k , what is X (t k , ζ )?
2 For a fixed sample point, ζ = ζ i , what is X (t , ζ i )?
3 For fixed t = t k & ζ = ζ i , what is X (t k , ζ i )?4 Be sure that you know the definitions of the following terms for
random processes: ensemble, member function, sample function,
realization
ECNG 6700 - Stochastic Processes, Detection and Estimation 16 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 17/35
Random Processes: Classification
ECNG 6700 - Stochastic Processes, Detection and Estimation 17 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 18/35
Random Processes: Classification
Stationarity:
Stationary (Strictly, Wide Sense)
Cyclostationary
Non–stationary
Real vs Complex–valued:
Real–valued bandpass RP - Z (t ) = A(t )cos [2πf c t + θ(t )]
Z (t ) =
{A(t )e j Θ(t )e j 2πf c t
}=
{W (t )e j 2πf c t
}Complex envelope,W (t ) = A(t )cos Θ(t ) + jA(t )sin Θ(t ) = X (t ) + jY (t )
ECNG 6700 - Stochastic Processes, Detection and Estimation 18 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 19/35
Random Processes: Methods of Description
Joint Distribution:F X (t 1),...,X (t n )(x 1, . . . , x n ) = P (X (t 1) ≤ x 1, . . . , X (t n ) ≤ x n )
Analytical Description using RVs:
Real–valued bandpass RP - Z (t ) = A(t )cos [2πf c t + θ(t )]
Average Values:
Mean - µX (t ) = E [X (t )]
Autocorrelation - R XX (t 1, t 2) = E [X ∗(t 1)X (t 2)]
Autocovariance - C XX (t 1, t 2) = R XX (t 1, t 2) − µX ∗(t 1)µX (t 2)
Correlation coefficient - r XX (t 1, t 2) = C XX (t 1,t 2)√ C XX (t 1,t 1)C XX (t 2,t 2)
ECNG 6700 - Stochastic Processes, Detection and Estimation 19 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 20/35
Random Processes: 2 or more RPs
Joint Distribution:F X (t 1),...,X (t n ),Y (t ‘1),...,Y (t ‘m )(x 1, . . . , x n , y 1, . . . , y m )= P (X (t 1) ≤ x 1, . . . , X (t n ) ≤ x n , Y (t ‘1) ≤ y 1, . . . , Y (t ‘m ) ≤ y m )
Analytical Description using RVs:
Real–valued bandpass RP - Z (t ) = A(t )cos [2πf c t + θ(t )]
Average Values:
Autocorrelation - R XY (t 1, t 2) = E [X ∗(t 1)Y (t 2)]
Autocovariance - C XY (t 1, t 2) = R XY (t 1, t 2) − µX ∗(t 1)µY (t 2)
Correlation coefficient - r XY (t 1, t 2) = C XY (t 1,t 2)√ C XX (t 1,t 1)C YY (t 2,t 2)
ECNG 6700 - Stochastic Processes, Detection and Estimation 20 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 21/35
Random Processes: 2 or more RPs
Properties:
For 2 uncorrelated RPs,
C XY (t 1, t 2) = 0, t 1, t 2 ∈ T
For 2 orthogonal RPs,
R XY (t 1, t 2) = 0, t 1, t 2 ∈ T
Independent
P [X (t 1) ≤ x 1, . . . , X (t n ) ≤ x n , Y (t ‘1) ≤ y 1, . . . , Y (t ‘m ) ≤ y m ]
=n
i =1{P [X (t i ) ≤ x i ]} m
j =1{P [Y (t ‘ j ) ≤ y j ]}
for all n , m & t 1, . . . , t n , t ‘1, . . . , t ‘m ∈ T
ECNG 6700 - Stochastic Processes, Detection and Estimation 21 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 22/35
Random Processes: Worked Examples
Example 2:
Let’s use MATLAB to define and investigate a Bernoulli (p ) randomprocess. . .
Example 3:
Consider the amplifier of a radio receiver. Because all amplifiers
internally generate thermal noise, even if the radio is not receiving anysignal, the voltage at the output of the amplifier is not zero but is well
modeled as a Gaussian random variable each time it is measured.
Suppose we measure this voltage once per second and denote the n th
measurement by Z n . Assume that the amplifier gain is 5 and an inputsignal, x (t ) = sin (2πft ), is applied.
Let’s investigate this further in MATLAB . . .
ECNG 6700 - Stochastic Processes, Detection and Estimation 22 / 35
Random (Stochastic) Processes
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 23/35
Special Random Processes
Markov Processes
Gaussian (Normal) Processes
Independent increments (e.g., Wiener Process, Poisson Process)
You should really just get a better idea of the key characteristics of
these processes!!!
ECNG 6700 - Stochastic Processes, Detection and Estimation 23 / 35
Stationarity
S i S i i (SSS)
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 24/35
Strict–sense Stationarity (SSS)
Definition:
1
The distribution function describing the process is invariant undera translation of time/space
2 For all t 1, . . . , t k , t 1 + τ , . . . , t k + τ ∈ T & all k = 1, 2, . . .,
P [X (t 1) ≤ x 1, . . . , X (t k ) ≤ x k ] = P [X (t 1 + τ ) ≤ x 1, . . . , X (t k + τ ) ≤x k ]
3 If stationary for k ≤ N but not k > N then X (t ) is a N th order
stationary process
SSS Properties:
Constant mean over index: E [X (t )] = µX = constant
Autocorrelation only depends on time difference, not actual time
samples: E [X ∗(t 1)X (t 2)] = R XX (t 2 − t 1)
Jointly SSS:
Joint distributions of X (t ) and Y (t ) are invariant to shifts in time
ECNG 6700 - Stochastic Processes, Detection and Estimation 24 / 35
Stationarity
Wid St ti it (WSS)
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 25/35
Wide–sense Stationarity (WSS)
WSS:
Less restrictive form of stationarity than SSSOnly based upon the mean and autocorrelation functions, µX (t )and R XX (t 1, t 2).
WSS Properties:
Constant mean over index: E [X (t )] = µX = constant
Autocorrelation only depends on time difference, not actual time
samples: E [X ∗(t 1)X (t 2)] = R XX (t 2 − t 1) or alternatively,
E [X ∗(t )X (t + τ )] = R XX (τ )
Jointly WSS:
E [X ∗(t )Y (t + τ )] = R XY (τ )
SSS implies WSS, but WSS does not imply SSS
ECNG 6700 - Stochastic Processes, Detection and Estimation 25 / 35
Stationarity
Oth F f St ti it
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 26/35
Other Forms of Stationarity
Asymptotical stationarity:
Distribution of X (t 1 + τ ), . . . , X (t n + τ ) does not depend on τ whenτ is large
Stationary on an interval:
X (t ) is SSS for all τ for which t 1 + τ, . . . , t k + τ lie in an interval
γ ⊂ T Stationary increments:
Increments X (t + τ ) − X (t ) form a stationary process for every τ
Cyclostationarity/Periodic stationarity:
X (t ) is stationary for a constant shift T 0, or integer multiples of T 0
Food for Thought:
When a complete description of X (t ) is unavailable, but some process
samples are available, how would you test for stationarity?
ECNG 6700 - Stochastic Processes, Detection and Estimation 26 / 35
Power Spectral Densities for Real WSS Processes
A t l ti F ti (ACF) f WSS P
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 27/35
Autocorrelation Function (ACF) of a WSS Process
Properties:1 Average power, R XX (0) = E [X 2(t )] ≥ 0
2 R XX (τ ) is an even function of τ : R XX (τ ) = R XX (−τ )
3
|R XX (τ )
| ≤ R XX (0)
4 If X (t ) contains a periodic component, then R XX (τ ) will also
5 If limτ →∞ R XX (τ ) = C , then C = µ2X
6 If R XX (T 0) = R XX (0) for some T 0 = 0, then R XX (τ ) is periodic with
period T 07 If R XX (0) < ∞ and R XX (τ ) is continuous at τ = 0, then R XX (τ ) is
continuous for every τ
ECNG 6700 - Stochastic Processes, Detection and Estimation 27 / 35
Power Spectral Densities for Real WSS Processes
Cross correlation Function of a WSS Process
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 28/35
Cross–correlation Function of a WSS Process
Properties:
1 R XY (τ ) = R YX (−τ )
2
|R XY (τ )| ≤ R XX (0)R YY (0)
3 |R XY (τ )| ≤ 12 [R XX (0) + R YY (0)]
4 R XY (τ ) = 0 if X (t ) and Y (t ) are orthogonal
5 R XY (τ ) = µX µY if X (t ) and Y (t ) are independent
ECNG 6700 - Stochastic Processes, Detection and Estimation 28 / 35
Power Spectral Densities for Real WSS Processes
Power Spectral Density (PSD) of a WSS Process
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 29/35
Power Spectral Density (PSD) of a WSS Process
Definition (Wiener-Khinchine Relation):
S XX (f ) = F{R XX (τ )} = ∞−∞ R XX (τ )e − j 2πf τ d τ
F{} is the Fourier transform operator
Autocorrelation can be retrieved using inverse transform on PSD
R XX (τ ) =
F −1
{S XX (f )
}= ∞−∞ S XX (f )e j 2πf τ df
Properties:
1 S XX (f ) is real & nonnegative
2 Average power in X (t ), E [X 2(t )] = R XX (0) = ∞−∞ S XX (f )df
3 For X (t ) real, R XX (τ ) is even and hence S XX (f ) is also even,S XX (f ) = S XX (−f )
4 If X (t ) has periodic impulses, then S XX (f ) will have impulses
ECNG 6700 - Stochastic Processes, Detection and Estimation 29 / 35
Power Spectral Densities for Real WSS Processes
Power & Bandwidth
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 30/35
Power & Bandwidth
Power in band [f 1, f 2],
P X [f 1, f 2] = f 2
f 1S XX (f )df +
−f 1−f 2
S XX (f )df = 2 f 2
f 1S XX (f )df
ECNG 6700 - Stochastic Processes, Detection and Estimation 30 / 35
Power Spectral Densities for Real WSS Processes
Cross power Spectral Density (CPSD)
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 31/35
Cross–power Spectral Density (CPSD)
Definition:
S XY (f ) = F{R XY (τ )} = ∞−∞ R XY (τ )e − j 2πf τ d τ
Cross–correlation can be retrieved using inverse transform on
CPSD
R XY (τ ) = F −1{S XY (f )} = ∞−∞ S XY (f )e j 2πf τ df
Properties:
1 S XY (f ) is generally complex valued
2 S XY (f ) = S ∗YX (f )
3
{S
XY (f
)}is an even function of f and
{S
XY (f
)}is an odd
function of f
4 S XY (f ) = 0 if X (t ) and Y (t ) are orthogonal
5 S XY (f ) = µX µY δ (f ) if X (t ) and Y (t ) are independent
ECNG 6700 - Stochastic Processes, Detection and Estimation 31 / 35
Power Spectral Densities for Real WSS Processes
RP Examples
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 32/35
RP Examples
Questions:
1 In a communication system, the carrier signal at the receiver is
modeled by X t = cos (2πft + Θ), where Θ ∼ Uniform [−π, π]. Find
the mean function and the correlation function of X t .
HINT: cosAcosB = 12 [cos (A + B ) + cos (A − B )].
2 Determine the stationarity of the above function.
3 What happens if the frequency or amplitude are random, as
opposed to the phase?4 What happens if more than one are random?
ECNG 6700 - Stochastic Processes, Detection and Estimation 32 / 35
Power Spectral Densities for Real WSS Processes
RP Examples
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 33/35
RP Examples
Questions:
1 Find the PSD of the RP with autocorrelation function,
R XX (τ ) =
1 − |τ |
T , |τ | < T
0, else
2 Find the PSD and effective bandwidth of the RP with
autocorrelation function, R XX (τ ) = Ae −α|τ |, A, α > 0
3 The PSD of a zero mean Gaussian RP is given by,
S XX (f ) =
1, |f | < 500Hz
0, else
Find R XX (τ ) and show that X (t ) and X (t + 1) are uncorrelated
and hence independent.
Uncorrelated Gaussian RVs are also independent!
ECNG 6700 - Stochastic Processes, Detection and Estimation 33 / 35
Conclusion
Conclusion
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 34/35
Conclusion
We covered:
Concluded Random Variable FundamentalsIntroduction to Random Processes
Recommended Reading:
Kay - Sections 9.1–9.3, 9.8, 11.8, 15.1–15.5, 16.1–16.7, 17.1–17.4,17.6–17.8
Your goals for next class:
Continue working with MATLAB and Simulink
Revise in class exercises based on today’s discussions and askquestions in the next class
Start HW3 and ask questions in the next class
Review notes on RPs Part II in prep for next class
ECNG 6700 - Stochastic Processes, Detection and Estimation 34 / 35
Q & A
Thank You
8/13/2019 Random Processes i
http://slidepdf.com/reader/full/random-processes-i 35/35
Thank You
Questions????
ECNG 6700 - Stochastic Processes, Detection and Estimation 35 / 35