1 Multivariate Normal Multivariate Normal Distribution Distribution Shyh-Kang Jeng Shyh-Kang Jeng Department of Electrical Department of Electrical Engineering/ Engineering/ Graduate Institute of Graduate Institute of Communication/ Communication/ Graduate Institute of Networking Graduate Institute of Networking and Multimedia and Multimedia
93
Embed
1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
11
Multivariate Normal DistributionMultivariate Normal Distribution
Shyh-Kang JengShyh-Kang JengDepartment of Electrical Engineering/Department of Electrical Engineering/
Graduate Institute of Communication/Graduate Institute of Communication/
Graduate Institute of Networking and Graduate Institute of Networking and MultimediaMultimedia
22
Multivariate Normal DistributionMultivariate Normal DistributionGeneralized from univariate normal Generalized from univariate normal densitydensityBase of many multivariate analysis Base of many multivariate analysis techniquestechniquesUseful approximation to Useful approximation to ““truetrue”” population distributionpopulation distributionCentral limit distribution of many Central limit distribution of many multivariate statistics multivariate statistics Mathematical tractableMathematical tractable
33
Univariate Normal DistributionUnivariate Normal Distribution
xexf
N
x 2/]/)[(
2
2
2
2
1)(
),(
44
Table 1, AppendixTable 1, Appendix
55
Square of DistanceSquare of Distance(Mahalanobis distance) (Mahalanobis distance)
)('
)())((
1
122
μxμx
xxx
66
pp-dimensional Normal Density-dimensional Normal Density
],,,['
vector random from sample a is
,,2,1,
2
1)(
),(
21
2/)()'(2/12/
1
p
i
p
p
XXX
pix
ef
N
X
x
Σx
Σμ
μxΣμx
77
Example 4.1 Bivariate NormalExample 4.1 Bivariate Normal
)1(
1,
),Corr(/
)Var(),Var(
)(),(
2122211
2122211
1112
1222
2122211
1
2221
1211
2122111212
222111
2211
ΣΣ
XX
XX
XEXE
88
Example 4.1 Squared DistanceExample 4.1 Squared Distance
22
22
11
1112
2
22
22
2
11
11212
22
11
11221112
22111222
2122211
2211
1
21
1
)1(
1],[
)('
xxxx
x
x
xx
μxΣμx
99
Example 4.1 Density FunctionExample 4.1 Density Function
]}2
[)1(2
1exp{
)1(2
1),(
22
22
11
1112
2
22
22
2
11
11212
2122211
21
xx
xx
xxf
1010
Example 4.1 Bivariate DistributionExample 4.1 Bivariate Distribution
11 = 22, 12 = 0
1111
Example 4.1 Bivariate DistributionExample 4.1 Bivariate Distribution
11 = 22, 12 = 0.75
1212
ContoursContours
pi
c
c
contourdensityyprobabilitConstant
iii
i
,,2,1,
:axes
at centered ellipsoidan of surface
)-()'(such that all
i
21
eΣe
e
μ
μxΣμxx
1313
Result 4.1Result 4.1
definite positive
for ),(1/for ),(
1
definite positive :
1
1-
1-
Σ
ΣeΣe
eeΣeΣe
Σ
1414
Example 4.2 Bivariate ContourExample 4.2 Bivariate Contour
]2
1,
2
1[',
]2
1,
2
1[',
rseigenvecto and seigenvalue
normal, Bivariate
212112
112111
2211
e
e
1515
Example 4.2 Positive CorrelationExample 4.2 Positive Correlation
1616
Probability Related to Probability Related to Squared DistanceSquared Distance
1y probabilit has
)()()'(
satisfying values of ellipsoid Solid21pμxΣμx
x
1717
Probability Related to Probability Related to Squared DistanceSquared Distance
1818
Result 4.2Result 4.2
),( bemust
every for )','(:'
)','(
:'
),(:
2211
ΣμX
aΣaaμaXa
Σaaμa
Xa
ΣμX
p
pp
p
N
N
N
XaXaXa
N
1919
Example 4.3 Marginal DistributionExample 4.3 Marginal Distribution
),(
:in ofon distributi Marginal
),()','(:'
','
'],0,,0,1['
),(:]',,,[
111
111
1
21
iii
i
pp
N
X
NN
X
NXXX
X
ΣaaμaXa
Σaaμa
Xaa
ΣμX
2020
Result 4.3Result 4.3
),(:
)',(:
),(:
11
2121
1111
ΣdμdX
AΣAAμAX
ΣμX
p
q
pqpq
pp
pp
p
N
N
XaXa
XaXa
XaXa
N
2121
Proof of Result 4.3: Part 1Proof of Result 4.3: Part 1
)',(:every for valid
))'('),('(:)('
))'()'(,)'((:)'(
'
,')(' ncombinatiolinear Any
AAΣAμAXb
bAAΣbAμbAXb
bAΣAbμAbXAb
bAa
XaAXb
qN
N
N
2222
Proof of Result 4.3: Part 2Proof of Result 4.3: Part 2
),(:
arbitrary is
)',''(:''
)','(:'
'')('
ΣdμdX
a
ΣaadaμadaXa
ΣaaμaXa
daXadXa
pN
N
N
2323
Example 4.4 Linear CombinationsExample 4.4 Linear Combinations
322211
2
33232213222312
13222312221211
32
21
3
2
1
32
21
3
, with verifiedbe can
)',(:
2
2'
110
011
),(:
XXYXXY
N
X
X
X
XX
XX
N
AAΣAμAX
AAΣ
Aμ
AX
ΣμX
2424
Result 4.4Result 4.4
),(:
|
|
,,
),(:
1111
2221
1211
2
1
)1)((2
)1(1
ΣμX
ΣΣ
ΣΣ
Σ
μ
μ
μ
X
X
X
ΣμX
q
qp
q
p
N
N
4.3Result in |Set :Proof))(()()(
qpqqqpq0IA
2525
Example 4.5 Subset DistributionExample 4.5 Subset Distribution
4424
2422
4
221
4424
242211
4
21
4
21
5
,:
,,
),(:
N
X
X
N
X
ΣμX
ΣμX
2626
Result 4.5Result 4.5
22
11
2
1
2
1
2222211111
1221
2221
1211
2
1
2
1
)(21
)1(2
)1(1
|'
|
,:
tindependen ),(:),,(: (c)
0 ifonly and ift independen:,
|
|
,:--- (b)
),Cov( t,independen :, (a)
21
21
2121
Σ0
0Σ
μ
μ
X
X
ΣμXΣμX
ΣXX
ΣΣ
ΣΣ
μ
μ
X
X
0XXXX
qq
qq
qq
qqqq
N
NN
N
2727
Example 4.6 IndependenceExample 4.6 Independence
) also and oft independen is (
tindependen are and
tindependennot :,
200
031
014
),(:
213
32
11
21
3
XXX
XX
X
XX
N
X
Σ
ΣμX
2828
Result 4.6Result 4.6
211
221211
221
2211
221
22
2221
1211
2
1
2
1
covariance
and )(mean with normal
is given ofon distributi lconditiona
0,
|
|
,),,(:
ΣΣΣΣ
μxΣΣμ
xXX
Σ
ΣΣ
ΣΣ
Σ
μ
μ
μΣμ
X
X
X
pN
2929
Proof of Result 4.6Proof of Result 4.6
22
211
221211
22
221
221211
12212
|
'|
'
covariance with normaljoint
:
)(
)(
,
|0
|
Σ0
0ΣΣΣΣ
AAΣ
μX
μXΣΣμX
μXA
I
ΣΣI
A
3030
Proof of Result 4.6Proof of Result 4.6
)),((
:given
),0(:)(
))()((
)
|)()((
)()(/),()|(t independen ,
tindependen are and )(
211
221211221
22121
221
211
221211221
221211
221
221211221
221211
22
221
221211221
221211
22221
221211
ΣΣΣΣμxΣΣμ
xXX
ΣΣΣΣμXΣΣμX
μxΣΣμxμXΣΣμX
xX
μxΣΣμxμXΣΣμX
μXμXΣΣμX
q
q
N
N
f
f
APBPBAPBAPBA
3131
Example 4.7 Conditional BivariateExample 4.7 Conditional Bivariate
)),(()|(
thatshow
),(:
22
212
112222
12121
2212
1211
2
12
2
1
xNxxf
Nx
x
3232
Example 4.1 Density FunctionExample 4.1 Density Function
]}2
[)1(2
1exp{
)1(2
1),(
22
22
11
1112
2
22
22
2
11
11212
2122211
21
xx
xx
xxf
3333
Example 4.7Example 4.7
)1(2/))(/(
21211
22121
2221211
2122211
22
222
2
2222
12112
1211
22
22
11
1112
2
22
22
2
11
11212
21211
222221211
)1(2
1
)(/),()|(
2)1(2)1(2
)(
2
1)(
)1(2
1
]2[)1(2
1
xxe
xfxxfxxf
xxx
xxxx
3434
Result 4.7Result 4.7
ondistributi theof percentile
)th (100upper thedenotes )( where
,1 is )}()()'(:{
ellipsoid solid theinsidey probabilit The (b)
: )()'( (a)
0),,(:
2p
2p
2p
1
2p
1
μXΣμXx
μXΣμX
ΣΣμX pN
3535
22 Distribution Distribution
)2,12/ withondistributi(Gamma
0,0
0,)2/(2
1)(
(d.f.) freedom of degrees :,
)1,0(:);,(:
,),,(:),,(:
2
22/12/22/2
2
1
2
2
2222
2111
2
ef
x
NX
ZNX
NXNX
i i
ii
i
iii
3636
22 Distribution Curves Distribution Curves
3737
Table 3, AppendixTable 3, Appendix
3838
Proof of Result 4.7 (a)Proof of Result 4.7 (a)
2
1
21
2
2
1
1
1
'
'
2'2
1'1
1
2
2
1
'
'
1
1
:)()'(),1,0(:
/
/
/
'
)',0(:)(,)(1
)()'(1
)()'(
p
p
iii
p
pp
iiii
pp
p
p
ii
p
ii
i
ii
p
i i
ZNZ
NZ
μXΣμX
Ieee
ee
e
e
e
AAΣ
AAΣμXAZμXe
μXeeμXμXΣμX
3939
Proof of Result 4.7 (b)Proof of Result 4.7 (b)
1)()()'(
by ddistribute
variablerandom new )()'(
),(:by ellipsoid the toassigned
yprobabilit theis)()'(
21
2
1
21
p
p
p
P
N
cP
μXΣμX
μXΣμX
ΣμX
μXΣμX
4040
Result 4.8Result 4.8
n
jj
n
jj
nn
n
j
n
jjjjpnn
jp
n
b
c
bbb
ccNccc
N
1
2
1
2
122112
1 1
222111
j
21
)()'(
)'()(
matrix covariancewith
normaljoint are and
)(,:
),(:
tindependenmutually :,,,
ΣΣcb
ΣcbΣ
VXXXV
ΣμXXXV
ΣμX
XXX
4141
Proof of Result 4.8Proof of Result 4.8
ΣAAΣ
ΣΣAAΣ
AAΣAμV
VAX
III
IIIA
Σ00
0Σ0
00Σ
Σ
μ
μ
μ
μ
ΣμXXXX
X
X
X
X
X
)(:' rmsofdiagonalteoff
)(,)(:' of termsdiagonalblock
)',(:,
,
),(:],,,['
1
1
2
1
2
22
1
21
21
2
1
''2
'1
n
jjj
n
jj
n
jj
pn
n
n
npn
bc
bc
Nbbb
ccc
N
4242
Example 4.8 Linear CombinationsExample 4.8 Linear Combinations
312123
22
21
321
1
34321
2223'
3'
)','(:'
201
011
113
,
1
1
3
),( identicalt independen:,,,
aaaaaaa
aaa
N
N
Σaa
μa
ΣaaμaXa
Σμ
ΣμXXXX
4343
Example 4.8 Linear CombinationsExample 4.8 Linear Combinations
0ΣVVXXXXV
ΣΣΣ
μμμ
ΣμXXXXV
V
V
VV
4
12143212
4
1
2
4
1
343211
)(),Cov(,3
201
011
113
)(
2
2
6
2
),(:2
1
2
1
2
1
2
1
1
1
11
jjj
jj
jjj
bc
c
c
N
4444
Multivariate Normal LikelihoodMultivariate Normal Likelihood
conclusion reach the tosmall toois )10( size sample However,
normality bivariatereject 50%an Greater th
39.1 with are nsobservatio 10 ofout Seven
39.134.4]4224,974.126[',
102927
309.62
128831.0003293.0
003293.0000184.0'
2927
309.62
39.1)5.0(
1030.1476.255
76.25520.005,10,
2927
309.62
2
221
5
2
1
2
12
22
5
n
d
dxx
x
x
x
xd
Sx
7979
Chi-Square PlotChi-Square Plot
)/)2
1(1()/)
2
1(( that Note
1 slope havingorigin the
throughlinestraight a resemble shouldplot The
)),/)2
1((( allGraph
freedom of degrees on with distributi square-chi
theof quantile /)2
1(100:)/)
2
1((
distance squared Order the
distance squared :)()(
2,
2)(,
,
2)(
2)2(
2)1(
12
njnjq
dnjq
p
njnjq
ddd
xxSxxd
ppc
jpc
pc
n
8080
Example 4.13 Chi-Square Plot Example 4.13 Chi-Square Plot for Example 4.12for Example 4.12
8181
Example 4.13 Chi-Square Plot Example 4.13 Chi-Square Plot for Example 4.12for Example 4.12
8282
Chi-Square Plot for Computer Chi-Square Plot for Computer Generated 4-variate Normal DataGenerated 4-variate Normal Data
8383
Steps for Detecting OutliersSteps for Detecting OutliersMake a dot plot for each variableMake a dot plot for each variableMake a scatter plot for each pair of Make a scatter plot for each pair of variablesvariablesCalculate the standardized values. Calculate the standardized values. Examine them for large or small Examine them for large or small valuesvaluesCalculated the squared statistical Calculated the squared statistical distance. Examine for unusually distance. Examine for unusually large values. In chi-square plot, large values. In chi-square plot, these would be points farthest from these would be points farthest from the origin.the origin.
8484
Helpful Transformation to Helpful Transformation to Near NormalityNear Normality
Original ScaleOriginal Scale Transformed ScaleTransformed Scale
Counts, Counts, yy
Proportions, Proportions,
Correlations, Correlations, rr
p
pp
ˆ1
ˆlog
2
1)ˆlogit(
y
r
rrz
1
1log
2
1)( sFisher'
p̂
8585
Box and Cox’s Box and Cox’s Univariate TransformationsUnivariate Transformations