ECE594I Notes set 4: More Math: Expectations of 1-2 R.V.'s · PDF fileECE594I Notes set 4: More Math: Expectations of 1-2 R.V.'s ... Random Variables, Random Signal Principles ...

ECE594I notes, M. Rodwell, copyrighted

ECE594I Notes set 4:More Math: Expectations of 1-2 R.V.'sp

Mark RodwellUniversity of California, Santa Barbara y

[email protected] 805-893-3244, 805-893-3262 fax


References and Citations:

DevicesState-Solid in Noise :Zielder Van PhysicsThermal : Kroemerand Kittel

:Citations / Sources

ng.EngineeriionsCommunicatofsPrinciple:Jacobs&Wozencraftory)(introduct s PrincipleSignal RandomVariables, Randomty, Probabili: PeeblesZ. Peyton

ive)comprehens(hard, Variables Randomandity Probabil:Papoulis

1982circaStanfordHellmanMartin:noteslectureyProbabilit1982circa Stanford, Cover, Thomas :notes lecture theory nInformatio

Designc ElectroniNoise Low :erMotchenbakng.EngineeriionsCommunicatofs Principle:Jacobs & Wozencraft

t dffS t d

circuits. in Noise :Notes nsApplicatio tor LinearSemiconduc National1982circa Stanford,Hellman, Martin:notes lecturey Probabilit

design)receiver (optical Personik & Smith noise), (device by Fukui Papers Kroemer and Kittel Peebles,Jacobs, & Wozencraft Ziel,der Van

study.for references Suggested

Theory nInformatio of Elements:Williams andCover )(!Notes App. Semi. National


random variablesrandom variablesandand

Expectationsp


Recall: Distribution Function of Random Variable

is and between lies y that probabilit The.valueparticulara ontakesvariablerandoma ,experiment an During

2

21 xxxxX

x

function.ondistributiy probabilit theis )(

)(}{1

21

xf

dxxfxxxP

X

xX∫=<<

yp)(f X

fX(x)

xx1 x2


Mean values and expectations

[ ] ∫∞+

dxxfxgxgE )()()(

X variablerandomtheofg(X)functiona of nExpectatio

[ ] ∫∞−

= dxxfxgxgE X

XfV lM

)()()(

[ ] ∫∞+

=== dxxxfXEXX X )(

X of Value Mean

∞−

X of valueExpected 2

[ ] ∫∞+

∞−

== dxxfxXEX X )(222


Variance

valueaverageitsfrom deviation square-mean-root its is X of varianceThe 2

xσ

( ) ( )[ ] ( ) )(

valueaverage its from

2222XX dxxfxxxXExXσ ∫

∞+

−=−=−=

i lifd i id dh

∫∞−

variance theofroot square thesimply isXofdeviation standard The xσ


Returning to the Gaussian Distribution

( ):ondistributi Gaussian thedescribing notation The

2 ⎞⎛ ( )2

exp2

1)( 2

2

2 ⎟⎟⎠

⎞⎜⎜⎝

⎛ −−

π=

xxX

xxxfσσ

clear.benow should

fX(x)

~2σx

xx


Variance vs Expectation of the Square

( ) ( )( )

( )22

22 xXxXxXX

−−=−=σ

( )

( )2

2 22

22

xXxX

xxXX

+⋅−=

+⋅−=

( )

( )2 22 xxxX +⋅⋅−=

( )square theof nexpectatio theis varianceThe

222 xXX

−=σ

n.expectatio theof square theminus


Example of Expectation: Mean Kinetic Energy

:ondistributivelocitya thermalwithparticleOur

/ where2

exp121)(

:ondistributivelocity a thermal withparticleOur

2

2

mkTvvf vx

xVx=

⎭⎬⎫

⎩⎨⎧−⋅= σ

[ ] 0)(

22 2

dvvfvvE V

vvx

==

⎭⎩

∫∞+

σσπ

[ ]

[ ] )(

0)(

22 dvvfvvE

dvvfvvE xxVxx x

= ∫

∫∞+

∞−

[ ]proof) skip---Gaussiana of variance thecomputes (this

)( dvvfvvE xxVxx x= ∫

∞−

[ ] [ ] 2/2/energyKineticSo

/ 2

2

kTmvEE

mkTv

==

== σ

[ ] [ ][ ] 2/

2/2/energy Kinetic So,kTEE

kTmvEE x

=


Example of Expectation: Shot Noise (Bernoulli Trial)

pqp

X⎩⎨⎧

−==

)1(y probabilit0y probabilit1

ppqxpxXE

ppqxxpXE

X

X

=⋅+⋅==

=⋅+⋅==

∑∑

2222 10)(][

10)(][

( ) pqppXEXE

ppqxpxXE

X

X

=−=−=

+∑2222 ][][

10)(][

σ


Example of Expectation: Quantization "Noise"

formlyuniddistributeR VanitselfisthatstipulateweIf AV

]2/2/[overddistributeuniformlyiserror onquantizati the then,2say range, large someover

formly uni ddistribute R.V.anitselfis that stipulate weIf

Δ+ΔΔ⋅±

ε

NAV

]2/,2/[over ddistributeuniformly is Δ+Δ−ε

withR.V.,analsotheniserror onquantizati The2/Δ

12][ ,0][

22/

2/

22 dEE εσεε εΔ

=Δ

=== ∫Δ+

Δ−

noise. aserror onquantizati treatingin cautious bemust wei.e. R.V.,an also is ifonly R.V.an is that Note AVε


Pairs of Random Variables

. variablesrandom of pairs understandfirst must we processes,random understand To

Yand X variablesrandom ofpair a ,experiment an In y.andx valuesparticular specific on takes

∫ ∫=<<<<D B

XY

XY

dxdyyxfDyCBxAP

yxf

),(} and {

),(ondistributijoint by thedescribedisbehavior joint Their

∫ ∫C A

XY yyfy ),(}{


Pairs of Random Variables

∫ ∫=<<∞+ B

dxdyyxfBxAP )(}{

definedbealsomust onsdistributi Marginal

∫

∫ ∫

=

=<<∞−

BA

XY

dxxf

dxdyyxfBxAP

)(

),(} {

∫=A

X dxxf )(

∫ ∫=<<∞+D

XY dxdyyxfDyCP ),(}{

:for Ysimilarly and

∫

∫ ∫

=

<<∞−

D

Y

CXY

dyyf

dxdyyxfDyCP

)(

),(} {

∫C

Y dyyf )(


Statistical Independence

wherecase theIn

t.independenlly statistica be tosaid are variablesthe,)()(),( yfxfyxf YXXY =

expected.generally not is This


Conditional Densities

fhilifihli R.V.y that theprobabilit lconidtiona heConsider t n.observatio

somemakingafter R.V.anofondistributi therevising in Arises

BX

)|(][

])[(]|)[(

.event ofoccurencethegivenvaluespecifica thanless is

| BxFBP

BxXPBxXP

BxX

BX=∩≤

≡≤

. given of function of ondistributi cumulative theis This][

BXBP

)|()|(:function onDistributi |/ BxFdxdBxf BXBX ≡


Conditional Densities (II)

? valueparticular some has that given of ndistibutio theis what ,),( variablesrandom ofpair a Given

yYXYX

),()|()|( , yxf

yxfyYxf YX===)(

)|()|( // yfyxfyYxf

YYXYX ===

)()|( then, If / xfyxfYX XYX =⊥


Expectations of a pair of random variables

[ ] ),(),(),(

isYandYvariablesrandomtheofY)g(X,function a ofn expectatio The

dxdyyxfyxgyxgE XY∫ ∫∞+ ∞+

=

: ofn Expectatio X

∞− ∞−

[ ] )(),( dxxxfdxdyyxxfxXE XXY ∫∫ ∫∞+

∞−

∞+

∞−

∞+

∞−

===

[ ] ofn Expectatio 2X

∫∫ ∫∞+∞+ ∞+

[ ] )( ),( 2222 dxxfxdxdyyxfxXXE XXY ∫∫ ∫∞−∞− ∞−

===

. and for similarly ...and 2YY


Correlation between random variables

[ ] )(

is Yand X of ncorrelatio The

d dfXYER ∫ ∫∞+ ∞+

[ ] ),(• dxdyyxfxyXYER XYXY == ∫ ∫∞− ∞−

( )( )[ ] [ ]is Yand X of covariance The

yxyXYxXYEyYxXECXY +−−=−−= ( )( )[ ] [ ] yxR

yxyXYxXYEyYxXEC

XY

XY

⋅−=+

valuesmeanzerohaveor YXeither if same theare covariance and ncorrelatio that Note

values.meanzero have or Y X


Correlation versus Covariance

component. varying- time thefrom bias) (DC valuemean theseparateusually wecurrents,andvoltageswith workingare weWhen

mean. zero have then variablesrandom The

analysisnoisecircuitincommonthereforeisIt

.covariance toequal thenis nCorrelatio

ably.interchang terms two theuse to analysisnoisecircuit incommon thereforeisIt

ons.distributi lconditionacalculate e.g. we whenreturn can valuesmean nonzero But,

careful. Be


Correlation Coefficient

/is Yand X oft coefficien ncorrelatio The

C σσρ

t i lif i( t d d)thN t

/ YXXYXY C σσρ =

.covariance and ncorrelatio betweengy terminoloinconfusion(standard) theNote


Sum of TWO Random Variables

: variablesrandom twoof Sum YXZ +=

[ ] [ ] [ ][ ] [ ] 2

2)(22

2222

XYRYEXEYXYXEYXEZE

++=

++=+=

[ ] [ ]

meanszerohavebothandIf

XY

YX[ ] [ ] [ ] 2

meanszerohaveboth and If222

XYCYEXEZEYX

++=

n.correlatio of role theemphasizes This


Uncorrelated Variables.

0:edUncorrelat

C

ti d dllSt ti ti

0CXY =

)()(),(:tindependenlly Statistica

yfxfyxf YXXY =

i d di ldl in.correlatio zero implies ceIndependen

ce.independenimply not doesncorrelatio Zero

ceindependenimply doeseduncorrelat s,JGRV'For


Summing of Noise (Random) Voltages

i t Rthtli dltT

[ ] 21)(1 P variablerandoma isresistor thein dissipatedpower The

resistor Rthetoapplied are voltagesTwo

222 VVVVVVPPE +++

V1 R

[ ]1211

2)(

22

21

22112

21

21V

RC

RV

R

VVVVR

VVR

PPE

VV ++=

++=+==

V2111

1211 22

21 21

VRR

VR

RRR

VV ++= σ

1211

2111

22

22

21 2121

VVVV

RV

RV

R VVVV

++=

++= σσρ

included.bemusttermoncorrellatia--addnot do generators random two theof powers noise The

2 2211 VR

VVR

VR

++=

add. do voltagesnoise random theof values timeousinstantane The

included.bemust termoncorrellatia


Shot Noise as a Random Variable

photonsreceivedof#thecallandphotononeSend.y probabilition transmisshasfiber The

Np

[ ] [ ] [ ] so and

.photonsreceivedof#thecall and photon, one Send

221

21

22111

1

1ppNNEpNEpNNE

N

N −=−==== σ

hi df#hllii d dlli iis each of sion transmis, them)of ( photonsmany sendnow weIf

1

NM

[ ] [ ] )(and

,photons received of#thecalling---sot,independenlly statistica

2221 ppMMMpNEMNE

N

NN −=⋅==⋅= σσ[ ] [ ]

,1 and ,1 ,1 supposeNow

)( and 1 1

MppM

ppMMMpNEMNE NN

>><<>>

σσ

count. theof valuemean theapproachescount theof varianceThe 2 NN =→σ


Thermal Noise as a Random Variable

T tureat tempera room) (a warm reservoir""a withmequilibriu in isresistor The.resistor Ra toconnected is Ccapacitor A

withmequilibriu thermalsestablisheit :power no dissipate can C heat. of form thein room thehenergy wit exchange can R

systemaoffreedomofdegreetindependenanyamicsthermodynFrom

resistor. via theroom the

2/hence kT/2,energy mean has T tureat tempera

systema offreedomof degreet independenany amics, thermodynFrom

kTE =

R

/

2/2/2

2

CkTV

kTCV

=

=

kT/C. variancehas voltagenoise The


Distribution of SumsDistribution of Sumsandand

Jointly Gaussian RV'sy


Distribution of a Sum of 2 Independent Random Variables

)(][)(

:variablesrandom*indendent* twoof Sum

dydxyxfzYXPzF

YXZYz

⋅⎟⎟⎞

⎜⎜⎛

=≤+=

+=

∫ ∫∞+ −

)()(

),(][)(

dydxxfyf

dydxyxfzYXPzF

Yz

XYZ

⋅=

⎟⎟⎠

⎜⎜⎝

=≤+=

∫∫

∫ ∫−∞+

∞− ∞−

so),()(But

)()(

zFdzf

dydxxfyf

ZZ

XY

=

⋅= ∫∫∞−∞−

so),()(But zFdz

zf ZZ

)(&)(ofnconvolutiothe)()()( zfyfdyyzfyfzf = ∫+∞

).(&)(ofnconvolutiothe,)()()( zfyfdyyzfyfzf XYXYZ −= ∫∞−


Example: Digital Transmission

NTRN

+=

signalreceivedon.distributi Gaussian noise, thermal

NTR +== signal received

not time) signal, ed transmittis ( )1()2/1()1()2/1()( ttttfT −++= δδ

2

exp2

1)( 2

2

2 ⎟⎟⎠

⎞⎜⎜⎝

⎛ −

π=N

nnfσσ 22 ⎠⎝π nnσσ


Example of Convolution of 2 Distributions: Communication

)(][)(

:variablesrandom*indendent* twoof Sum

dydxyxfzYXPzF

YXZYz

⋅⎟⎟⎞

⎜⎜⎛

=≤+=

+=

∫ ∫∞+ −

)()(

),(][)(

dydxxfyf

dydxyxfzYXPzF

Yz

XYZ

⋅=

⎟⎟⎠

⎜⎜⎝

=≤+=

∫∫

∫ ∫−∞+

∞− ∞−

so),()(But

)()(

zFdzf

dydxxfyf

ZZ

XY

=

⋅= ∫∫∞−∞−

so),()(But zFdz

zf ZZ


Distribution of a Sum of Many Independent RV's

)(&)(ofncon ol tiothe)()()(Gi en ffdfff ∫+∞

),(&)(ofnconvolutiothe,)()()( Given zfyfdyyzfyfzf XYXYZ −= ∫∞−

onsdistributiuniformidentical ...1,2,4,convolving that see can weGaussian.a similar to forma toleadslowly will

, , ,g

rem.limit theo central theof sense crude some gives This


Distribution of a Sum of a Few Random Variables

i bld*i d dh*fS YXZ

)()(and ),(][)(

:variablesrandom*indendentnot perhaps * twoof Sum

ZZ

Yz

XYZ zFddzfdydxyxfzYXPzF

YXZ

=⋅⎟⎟⎞

⎜⎜⎛

=≤+=

+=

∫ ∫∞+ −

dGid it&i itfdiffi ltjiThi

)()(),(][)( ZZXYZ dzfyyf ⎟

⎠⎜⎝∫ ∫∞− ∞−

)2()1()0()(filterlinear a and ),( time)of function (random process

randoma Given design. system&circuit for difficultymajor a is This

in

tVatVatVatVtV

+++ τττ

integralsnconvolutiocomputingrequiresfunction ondistributi its find toand , variablesrandom of suma is output the

...)2()1()0()( 210

out

inininout

VtVatVatVatV +⋅−+⋅−+⋅−= τττ

.difficultythisavoid(next)variablesrandomGaussianJointly

integrals.nconvolutiocomputing requires function

.difficultythisavoid(next)variablesrandom GaussianJointly


Pairs of Jointly Gaussian Random Variables

XY yxf 1),(

:GaussianJointly are Yand X If

=XYYX

XY

yyyyxxxx

yxf

)())(()()1(2

1exp

12),(

2

2

2

2

2

2

⎥⎤

⎢⎡

⎟⎟⎠

⎞⎜⎜⎝

⎛ −+

−−+

−⋅−×

− ρσπσ

YYXXXY

variablesof#largeratoextendedbecandefinitionThis

)1(2p 222 ⎥

⎦⎢⎣

⎟⎠

⎜⎝− σσσσρ

. variablesof#larger a toextendedbe can definition This

nXXXofsetabyspecifiediswhich

),,,( vector random GaussianJointly a have can wegeneral, In

21 L

[ ] [ ]jiiii xxExxEx scovariance and , variances, meansofset a by specified is which


Linear Operations on JGRV's

and define weif and Gaussian,Jointly are Yand X If

dYcXWbYaXV +=+=Gaussian.Jointly also are and Then

and WV

dYcXWbYaXV +=+=

functionsGaussian2ofnconvolutiobecause arisesresult theproof; without stated is This

function. Gaussiana produces functionsGaussian2 of nconvolutio

number.any of JGRVsfor holdsresult The


Probability distribution after a Linear Operation on JGRV's

[ ] [ ][ ] [ ] [ ] [ ] +−⋅++=−=

+=+=+==22222222 )(2

and

V YbXaXYEabYEbXEaVVEYdXcWYbXabYaXEVEV

σ

ted[ ] [ ] [ ][ ] [ ] [ ] [ ][ ] [ ][ ]

−++=−=

+−⋅++=−= 22222222

))(()(2

VW

W

WVdYcXbYaXEWVVWECYdXcXYEcdYEdXEcWWEσ

dious det

[ ][ ] [ ] [ ] ++−⋅+++=

−+++=22

22

))(()( )(

YdXcYbXaYEbdXYEbcadXacEWVbdYXYbcadacXE

tails

1)(

. and of ondistributijoint thecalculatenow can We

wvf

WV

⎥⎤

⎢⎡

⎟⎟⎞

⎜⎜⎛ −

+−−

+−

⋅−×

−=

22

2

)())(()(1exp

12),(

VWWVVW

wwwwvvvv

wvfρσπσ

⎥⎦

⎢⎣

⎟⎟⎠

⎜⎜⎝

++⋅−

−× 222 )1(2exp

WWVVVW σσσσρ


Jointly Gaussian Random Variables in N Dimensions

andvector)random(avariablesrandomofsetaDefine X:matrix covariancea

and vector)random(a variablesrandom ofset a Define

XCX

⎤⎡⎤⎡ CCCX

( )( )[ ]...

2212

12111

2

1

X XXXXCX⎥⎥⎥⎤

⎢⎢⎢⎡

=−−=⎥⎥⎥⎤

⎢⎢⎢⎡

=

N

XXXX

XXXXXX

T CCCCC

EXX

OMM

ts.coefficienncorrelatioofmatrixais1

XC

⎥⎥

⎦⎢⎢

⎣⎥⎥

⎦⎢⎢

⎣ NNN XXXXN CCXOMM

ts.coefficienncorrelatio ofmatrix a is XC

( ) ( )( ) ( )⎟

⎠⎞

⎜⎝⎛ −−⋅−= − XXCXX

Cx X

XX

12/12/ 2

1exp]det[2

1)( TNf

π( ) ( ) ⎠⎝X ][

, s RV'Gaussianjointly on operationslinear :pointKey . withs, R.V.'Gaussianjointly tolead , as such T

XY LLCCLXY ==


Why are JGRV's Important ?

:conclusionclear a is but theretediouswasslidelast theon math The

idl if track keep tosufficient isit ,operationslinear tosubjected sJGRV' With

found.simplybealwayscanfunctionsondistributin,informatiothisWith

.variancesandns,correlatio means, of

systemslinear in npropagatio noise of nscalculatio simplifies vastly This

found.simply bealways canfunctionsondistributin,informatio thisWith

circuits).(linear


Linear Filtering Operations

)( points at time samples itsby determineduniquely iswaveformdbandlimitea theorem,sampling sNyquist' From

⋅n τ

timediscrete infilter linear a analyze thereforecan We

: vectorsas signals the writeWe...)2()1()0()( 210 +⋅−+⋅−+⋅−= inininout tVatVatVatV τττ

],...,,[...

],...,,[)](),...,1(),0([

,21

21

in

out

V

V

==

=⋅⋅⋅=T

Nininin

ToutNoutout

ToutNoutout

VVV

VVVnVVV τττ

ns.nsformatiolinear tra areFiltersvectors.are waveformsTime window, timefinite some in Hence inout MVV =


Linear Filtering Operations

formed. are s R.V.'of Sums ns.nsformatiolinear traundergoes systems & circuitslinear throughgpropagatin Noise

variances&meanscalculateonlymustweHenceGaussianjointly also are s R.V.'filtered the thenGaussian,Jointly are s R.V.' theIf

ons.distributiy probabilit determine toscovariance andvariances&meanscalculateonly must we Hence,Gaussian.jointly

integrals.nconvolutiobydeterminedbemustsR.V.'filtered theof onsdistributiy probabilit Gaussian,jointly not are s R.V' theIf

randommany causes remlimit theocentral that thefortunate isIt

integrals.nconvolutioby determined bemust s R.V.filtered

Gaussian.jointly be toprocessesy


EstimationEstimation


Conditional Densities Again

)(f)(

),()|()|( ,

// yfyxf

yxfyYxfY

YXYXYX ===

)|()(]|)([

on ExpectatilConditiona

dxyxfxgyXgE ∫∞+

⋅≡

: Yof valueparticulara given X of nExpectatio

)|()(]|)([ / dxyxfxgyXgE YX∫∞−

≡

)|(]|[ / dxyxfxyXE YX∫∞+

∞−

⋅≡

ti tifti t. valueon takenhas RVthe that

observedhavethat wegiven,RVtheof valueted ExpecyY

X

.estimationfor nt ...importa


Estimation

.ˆ valueits estimate to wish we, RVan Given YY →

222 ˆ][2ˆ][])ˆ[(M.S.E.

error square-mean :estimate thisofquality of measure Possible

YYEYYEYYE ⋅−+=−=

:MMSEEestimateerror squared mean Minimum

][2][])[(M.S.E. YYEYYEYYE +

2 ])ˆ[( minimize topicked ][ˆq

YYEYEY −=

222 ][][2])[(][MMSEEerror MMS squared mean Minimum

YYEYEYEYE σ=⋅−+=


Estimation

ib ihidhbf

bhifiMMSE)(ˆ then, RV)correlatedy (presumabl theof valuethe

estimatetonobservatiothisuseand, R.V. theobserve weIf

XYY

YX

]|[)(ˆ

.observe that wegivenof estimate MMSE)(

YEY

xXYxY ==

]|))(ˆ[(MMSE

]|[)(22

| xYxYE

xYExY

xY −==

=

σ


Estimation with JGRV's

YX :simplifiesanalysistheandsJGRV'ofpairaFor

( )YyCXyXE

YX

Y

XY −⋅+= 2]|[

:simplifies analysistheandsJGRV ofpair a For

σ

ECE594I Notes set 4: More Math: Expectations of 1-2 R.V.'s · PDF fileECE594I Notes set 4: More Math: Expectations of 1-2 R.V.'s ... Random Variables, Random Signal Principles ...

Documents