Stanford University—April 28, 1997 Flexible Discriminant and Mixture Models 1 Flexible Discriminant and Mixture Models Trevor Hastie [email protected]Statistics Department and Division of Biostatistics Stanford University joint with Andreas Buja and Rob Tibshirani April 28, 1997 Papers fda.ps.Z, pda.ps.Z and mda.ps.Z are available from: ftp://playfair.stanford/edu/pub/hastie
32
Embed
Flexible Discriminant and Mixture Models - Stanford University
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Stanford University—April 28, 1997 Flexible Discriminant and Mixture Models 1'
where S = linear regression, additive regression, MARS,
NN,. . . , each giving a different version of FDA.
Stanford University—April 28, 1997 Flexible Discriminant and Mixture Models 13'
&
$
%
FDA and Penalized Discriminant Analysis
The steps in FDA are� Enlarge the set of predictors X via a basis expansionh(X), and hence inject us into a higher dimensional
space.� Use (penalized) LDA in the enlarged space, where the
penalized Mahalanobis distance is given byD(x; �) = (h(x)�h(�))T (�W+)�1(h(x)�h(�))�W is defined in terms of bases functions h(xi).� Decompose the classification subspace using a
penalized metric:max tr(UT�BetU) subject to UT (�W +)U = I
Stanford University—April 28, 1997 Flexible Discriminant and Mixture Models 14'
&
$
%
Skin of the Orange
x[,1]
x[,2
]
-4 -2 0 2 4
-4-2
02
46
00
00 00
000
0
0
00
00
00 0
00
0
00
0
0
000 0
0
0
0
00
0
000
0 0000
0
0
0
00
0
00
0
0
00
0
0
000
00
0
00 0
0
0
0
000
0
000
00 0
0 0
0 0
0
000
00 0
000
0
0
00 00
0
00
0000 0
0
000
0
0
0
0
0
0
00
0
00
0
0 000
000
00
00 0
0
0
000 00
0
00
0 0 0
00
00 0
0
00 0
0
00 0
0
00
0
00
0
0
0
0
0
00 0
0
0
000
0
0 00
00
0
00
00
00
00
0
0
00
0
0
0
0
0
0
0
0
0
0
0
000
0
0
0
000
0
0
0
0
0 0
0
0
0 0
0
0
0
000
0
0
0
0
0
0
0
0
0
00
0
0
0
0
00
0
0
0
00
00
0
00
0
000
0
0
0
0
0
0
0
0
0
00
00
0
0
0
0 0
0
0
0
00
0
0
0
0
0
00 00
0
0
0
0
00
000
00
0
0
00
0
0
0 00 00
0
0
00
0
0
0
00
0
0
0
0
00
0
0
00
0
0
0
00
0
0
0
0
0
0
0
00
00
0
0
0
00
0
0
0
0 000
0
0
0
0
0
0
0 0
0
0
0
0
00
0
0
0
00
0
00
0
00
00
0
0
00
00
Training Data
x[,1]
x[,2
]
-4 -2 0 2 4
-4-2
02
46
00
00 00
000
0
0
0
00
00 0
00
0
00
0
000
0
00
0
000
0 0000
0
0
00
00
0
00
0000
00
0
00 0
0
0
0
000
0
000
00 0
0 0
0 0000
00 0
000
0
00 00
0
00
0000 0
000
0
0
0
0
0
0
00 000 000
00
00
00 0
0
0
00 00
00
0 0 0
00 0
0
00 0
00 0
0
00
0
00
0
0
0
0
0
00 0
0
000
0
0 00
00
00
00
0
000
00
0
0
0
000 00 0
00
0
0
00 0
0
00
000
00
0 0
0
0
0
0
0 000
0
0
00
0
0
000
0
0
0
0
0
00
00
00
00
00 000
0
0
0
00
0
000
0
00
0 00
0
0
0
0
000
0
000
0
00
0
0
0
00
0
0
00
0 0
00
00
0
00
0
0
0
00
0
0
0
0
0 0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
00
00
0
00
0
000
0 0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
000
0
0
0
0
0 00 00
0
00
0
0
0
0
0
00
0
0
00
0
0
0
0
0
0
0
000
0
0
0
000
0
0
0
0
0
0
0
0
00
0
00
0
00
Predicted Classes
Stanford University—April 28, 1997 Flexible Discriminant and Mixture Models 15'
&
$
%
FDA vs Regression
In FDA algorithm, we decomposeY TS(�)Y = Y T YY fitted values for dummy response matrix
Why not stop at Y � E(Y jX)?i.e. for new x, computey1(x); y2(x); : : : ; yJ(x)and assign x to the class j with the largest yj(x).
Stanford University—April 28, 1997 Flexible Discriminant and Mixture Models 16'
Stanford University—April 28, 1997 Flexible Discriminant and Mixture Models 22'
&
$
%PDA: Coordinate 1
PD
A: C
oord
inat
e 2
0
0
0
0
0 0
0
00
0
00
0 0
000
0
0
0
00
0
0
00
00
0 00
0 0
00
0
0
000
0
00
0
0
0000
00
0
000
0 0
0
00
0
0
0
0
0
0
00
0
0
000
0
0
00
0
0 00
0
00
0 0000
00 0
00
0
0
00
00
0 000
0
0
0
0
0
0 00
00
0
0
0 0
00
0
0
0
0
0
00
0
0
0 0
0
0
0
00
0
0
0
0
0
0
0
0
0
000
0
00
000
0
0
0 0
0 0
0
00 00
0
0
0 0
0
0
0
0
0 0
00
0
0
0
0
0
0
0
00 00
0 0
0
0
0
000
0 00
00
0 0
00 000
000
0
0
0
0
0
0
0
0
00
0
0
0
00
000
00
0
0
0
0
0
00
0
0
0
0
0
00
0
0 0
0
000
00
00
0
0
0
0
0
00
0
00 0
00
0
0
0
0
00
0
0
0
0
0
0
0
0
0
0
0
0
0
00 0
000
1
1
1111
11
1
1
11
1
1
111
1
1
11 111
1
1
1
1
1
1
1
11
11
111
1
1111 1
1
1
11
11
1 111
11
11
1
11
1111 11
1 1
1
1 1111
11
1
11
11
1
1
1
1
11
1
11111
1
11 111
11
1
1111
11
1
111
1
1111
11
1
1
1
1 111
1
1
1
1
1
1
11
1
11
1
111
1
1111 1
11
11
11111
1
111
1
11
1
1
11 11
1
1 11111
1
11
11
1
1
11111
1
11 1
11
11 1111
1 1
11
1 11
1
1
1111
1
1
11
1
111
111
111 1
111 11
111
1111 1
11
111
111
1
1 1
1
1
1
1111
1
1
111
3
3
33
333
3
333
33
3
3
3
3
3
333
3
3
3333 3 3
33
33
3
33
3
3
3
3
333
3
3
3
3 33
3
3
333
33
3 3
3
3 3333
3 33
3
3
3
3
3
3 333
3
3
33
3
3
3
3
3 3
3
3
3
3
3
3
3
3
33
3
333
3 33
3
33
3
3
3
3
3
3 3
3
3
3
3
33
33
3
33
3
3
3 3
3
33
3
3
3
3
3
3
333
33
3
33
33
3
3
3
33
3
33
3
3
33
3
3
3
3
3
3
3
3
3
3
33
3
3
33
3333
3
3
3
3
3
33
3
3
3
3
33
333
3
3
3
3
3
3
3
3
3
3
3
3
3
3
33
3
33
3
3
3
3
3
3
333
33
3
3
3
3
3
3
3
3
3
33
33
3
3
333
3
3
33
3
3
333
3
9
999
9
9
9
9
9
9
9
9
9
9
9
9
99
99
99
99
9
9
99 9 9
9
9
9
9
9
9
9
9
9
99
99 99
9
9
9
999
9
9
9 9
9
9
9
9
9
9
9
9
99
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
999
9
99
9
9
9
99
9
9
9
9
9
99
9
9
9
9
9 9
9
9
9
9
9
99
9
99
9
9
9
9
9 999
99
99 9
9
99
999
9
9
9
99
9
9
9
9
99
9
9
9
9
999
99
9 9
9
9
9
8
8
8
8
88
8
8
88
8
8
8
8
8 8
8
88
8
88
8
8
8 8
8
8
8
8
8
8
8
8
88
8
8
8
8
8
88
8
88
8
88
88
8
8
8
8
8
8
88
8
8
8888 8
8
88
8
8
8
88
8
8
88
8
8
8
8
88 8
8
8
88
8
8
888
8
88 8
8
8
88 8
8 8
8
8
8 88
88
8
88
8
8
88
88
8
8
8
8 8
8
8
8
88
8
8
8
8
88
88
8
8
88
8
8
88
88 8
8
8
8
88
8
8
8
88
8
8
8
8
8
8
8 8
4
4
4
4
4
44
4
4
4
4
4
444
4
4
4
4
4
44
44
44
4
4
4
4
4
4
44
4
44
444
4
44
4
4
4
4 4
4
44
44
4
4
4 44
44
4
4 4
4
44
4
4
444
44
44
4
444
4
4 4
4
4
4
44
44
4
4
44
4 44
4
4 4
4
4
4 4 44
4
4
44
4
4
4
44
4
4
44 44
4
4
4
44
44
4 44
4
4
4
4
44
4
4 44
4
4
4
4
4
4
4
4
4
44
4 44
4
4
6
6
66
6
666
66
66
66
6
666
6
66
66
6
6
666
6
6
6
6
6
6
6
6
6
66
6
66
66
6
6
66
6
6
66
6
6
6
66
6
66
66
6 6
66
6
6
6
6
66
6
6
6
6
6 6
6
6
6
6
6
66
6
66
66
666 6
6
6
6
66
6
6
6
6
6
6 6
66
66
6
66
6
6
66
6
6
6
6
666
6
66 6
66
6
6
6
6
6
6
6 66
6
66
66
6
6
66
6
6
6
6
6
6 6
6
6
66
6
6
66
6 6
6
6
6
666
666
66
6
6
66
6
66
6
66
6
6
6
6
6
6
6
6
6
6
6
6666 6
6
6
6
6
6
66
6
6
6
66
6
6
6
6
6
6
7
77
77
7
7
77
7
7
7
77
77 77
77
7
77 7
7
77
7
7
7
77
7
7
7
77
7
7
77
7
7
7
7
77
7
77
7 77
77
7
7
7
7 77
7
77
77
77 7
7
7
7
77
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
77
7 7
777
77
7
77
77
7
777
7
7
7
7
77
77
777
777
7
7
77
77
77 7
7
7
7
7
7777 7
777
77777 7
7 7
7
77
77
77
77
7
7
7
7
7
7
7
7
7
77
7
7
7
7
7
7777
7 77
7
7
7 77
5
5
5
5
5
5
5
55
5
55
5
5
5
5
5 5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
555 55
5
5
5
5
5
5 55
5
55
5
55
5
55
5
55
5
5
5
55
5
55
55
5
55
5
5
5
55
5
5
5
5
55
5
5
55
5
5
5
5555
5
55
5
5 5
5
55
5
5
5
5
5 5
5
5
5
5
5
5
5
55
5
5
5
5
5 5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
55
55
55
5
5
55
5
5
5
5
55
5 55
52
22
2
22
22
22
2
2
2
2
2
2
22
22 2
2
2
2
2
2
22
2
2
2
2
2
2
22
22
2
22
2
2
2
2
22
22
2
2
2
2
2
2
22
2
2
2 2
22
22
2
2
22
22 2
22 2
2
2
2
22
22
22
2
2
2 2
22
22 22 2
22
2
2
2
2
222
2
2
22
2 2
222
222
22
2
2
Canonical Variate Plot --- Digit Test Data
Stanford University—April 28, 1997 Flexible Discriminant and Mixture Models 23'
&
$
%
PDA coefficients: Phoneme Data
Frequency
Can
onic
al F
unct
ion
0 50 100 150 200 250
-0.2
-0.1
0.0
0.1
0.2
order 1
Frequency
Can
onic
al F
unct
ion
0 50 100 150 200 250
-0.2
-0.1
0.0
0.1
0.2
order 2
Frequency
Can
onic
al F
unct
ion
0 50 100 150 200 250
-0.2
-0.1
0.0
0.1
0.2
order 3
Frequency
Can
onic
al F
unct
ion
0 50 100 150 200 250
-0.2
-0.1
0.0
0.1
0.2
order 4
Ordinary LDA coefficient functions for the phoneme data,
and regularized versions.
Stanford University—April 28, 1997 Flexible Discriminant and Mixture Models 24'
&
$
%
Mixture Discriminant Analysis: MDA
1
1
1 1
1
11
1 1
11
1
1
1
1
1
1
1
11
1
1
1
111
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
1
111
1
1
1
1
1
11 1
1
1
1
1
1
1
11
1
1
1
1
1
1 1
1
1 11
1 111
1
1 11
1
1
11 1
1
1
1
1
1
11
11 1
11
1
11 11
1
1
1
1
11
1
1
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
11
1
1
1
1
1
1
1
1
11
11
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1 11
11
1
1
1 1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
11
1
1
11 1
11 111
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
1 1 1 11
1
1
1
1
11
1
1
1
111
1
1
11
1
1
11
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
222
2
2
2
2 2
2
2
22
222
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
2
2
2
22
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2 2
22
2 22
2 22
2
2 2
2
2
2
2
2
2
2
2
2
2
2
22
2
2 2
2
2
22
2
2
22
2 2
2
22
2
22
22
2
2
2
2
2
2 2
2
2
22
2
2
2
2
2
22
2
2
2
2
2
2
2
2
2 2
2
222
2
2
2 2
2
22
2
222
22 2
2
2
2
2 2
2
2
2
2
2
2
2
2
2
2
2
2
2
2222
2 2
2
2
2
Linear Discriminant Analysis
1
1
1 1
1
11
1 1
11
1
1
1
1
1
1
1
11
1
1
1
111
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
1
111
1
1
1
1
1
11 1
1
1
1
1
1
1
11
1
1
1
1
1
1 1
1
1 11
1 111
1
1 11
1
1
11 1
1
1
1
1
1
11
11 1
11
1
11 11
1
1
1
1
11
1
1
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
11
1
1
1
1
1
1
1
1
11
11
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1 11
11
1
1
1 1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
11
1
1
11 1
11 111
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
1 1 1 11
1
1
1
1
11
1
1
1
111
1
1
11
1
1
11
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
222
2
2
2
2 2
2
2
22
222
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
2
2
2
22
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2 2
22
2 22
2 22
2
2 2
2
2
2
2
2
2
2
2
2
2
2
22
2
2 2
2
2
22
2
2
22
2 2
2
22
2
22
22
2
2
2
2
2
2 2
2
2
22
2
2
2
2
2
22
2
2
2
2
2
2
2
2
2 2
2
222
2
2
2 2
2
22
2
222
22 2
2
2
2
2 2
2
2
2
2
2
2
2
2
2
2
2
2
2
2222
2 2
2
2
2
Mixture Discriminant Analysis
••
•
1
1
1 1
1
11
1 1
11
1
1
1
1
1
1
1
11
1
1
1
111
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
1
111
1
1
1
1
1
11 1
1
1
1
1
1
1
11
1
1
1
1
1
1 1
1
1 11
111
11
1 1
11
1
11 1
1
1
1
1
1
11
11 1
11
1
11 11
1
1
1
1
11
1
1
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
11
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
1 1
11
1
1
1
1
1
1
1
1
11
11
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1 11
11
1
1
1 11
1
1
1
1
1
1
1
1
11
1
1
1
1
1
11
1
1
11 1
11 111
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
1 1 1 11
1
1
1
1
11
1
1
1
111
1
1
11
1
1
11
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
222
2
2
2
2 2
2
2
22
222
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
2
2
2
22
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
22
22
2 22
2 22
2
2 2
2
2
2
2
2
2
2
2
2
2
2
22
2
2 2
2
2
22
2
2
22
2 2
2
22
2
22
22
2
2
2
2
2
2 2
2
2
22
2
2
2
2
2
22
2
2
2
2
2
2
2
2
2 2
2
222
2
2
2 2
2
22
2
222
22 2
2
2
2
2 2
2
2
2
2
2
2
2
2
2
2
2
2
2
2222
22
2
2
2
Learning Vector Quantization
••
••
•
•
•
1
1
1 1
1
11
1 1
11
1
1
1
1
1
1
1
11
1
1
1
111
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
1
111
1
1
1
1
1
11 1
1
1
1
1
1
1
11
1
1
1
1
1
1 1
1
1 11
111
11
1 1
11
1
11 1
1
1
1
1
1
11
11 1
11
1
11 11
1
1
1
1
11
1
1
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
11
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
1 1
11
1
1
1
1
1
1
1
1
11
11
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1 11
11
1
1
1 11
1
1
1
1
1
1
1
1
11
1
1
1
1
1
11
1
1
11 1
11 111
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
1 1 1 11
1
1
1
1
11
1
1
1
111
1
1
11
1
1
11
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
222
2
2
2
2 2
2
2
22
222
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
2
2
2
22
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
22
22
2 22
2 22
2
2 2
2
2
2
2
2
2
2
2
2
2
2
22
2
2 2
2
2
22
2
2
22
2 2
2
22
2
22
22
2
2
2
2
2
2 2
2
2
22
2
2
2
2
2
22
2
2
2
2
2
2
2
2
2 2
2
222
2
2
2 2
2
22
2
222
22 2
2
2
2
2 2
2
2
2
2
2
2
2
2
2
2
2
2
2
2222
22
2
2
2
Flexible Discriminant Analysis
1
1
1 1
1
11
1 1
11
1
1
1
1
1
1
1
11
1
1
1
111
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
1
111
1
1
1
1
1
11 1
1
1
1
1
1
1
11
1
1
1
1
1
1 1
1
1 11
111
11
1 11
1
1
11 1
1
1
1
1
1
11
11 1
11
1
11 11
1
1
1
1
11
1
1
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
11
11
1
1
1
1
1
1
1
1
11
11
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1 11
11
1
1
1 1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
11
1
1
11 1
11 111
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
1 1 1 11
1
1
1
1
11
1
1
1
111
1
1
11
1
1
11
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
222
2
2
2
2 2
2
2
22
222
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
2
2
2
22
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2 2
22
2 22
2 22
2
2 2
2
2
2
2
2
2
2
2
2
2
2
22
2
2 2
2
2
22
2
2
22
2 2
2
22
2
22
22
2
2
2
2
2
2 2
2
2
22
2
2
2
2
2
22
2
2
2
2
2
2
2
2
2 2
2
222
2
2
2 2
2
22
2
222
22 2
2
2
2
2 2
2
2
2
2
2
2
2
2
2
2
2
2
2
2222
2 2
2
2
2
Mixture Discriminant Analysis
Rank 1 Model
••
•1
1
1 1
1
11
1 1
11
1
1
1
1
1
1
1
11
1
1
1
111
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
1
111
1
1
1
1
1
11 1
1
1
1
1
1
1
11
1
1
1
1
1
1 1
1
1 11
111
11
1 11
1
1
11 1
1
1
1
1
1
11
11 1
11
1
11 11
1
1
1
1
11
1
1
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
11
11
1
1
1
1
1
1
1
1
11
11
1
1
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1 11
11
1
1
1 1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
11
1
1
11 1
11 111
1
11
1
1
1
1
1
1
1
1
1
1
1
1
1
11
1
1
1 1 1 11
1
1
1
1
11
1
1
1
111
1
1
11
1
1
11
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
222
2
2
2
2 2
2
2
22
222
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2
2
2
2
22
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
22
2 2
22
2 22
2 22
2
2 2
2
2
2
2
2
2
2
2
2
2
2
22
2
2 2
2
2
22
2
2
22
2 2
2
22
2
22
22
2
2
2
2
2
2 2
2
2
22
2
2
2
2
2
22
2
2
2
2
2
2
2
2
2 2
2
222
2
2
2 2
2
22
2
222
22 2
2
2
2
2 2
2
2
2
2
2
2
2
2
2
2
2
2
2
2222
2 2
2
2
2
MDA/FDA
Stanford University—April 28, 1997 Flexible Discriminant and Mixture Models 25'
&
$
%
Gaussian Mixture Model
P (XjG = j) = RjXr=1 �jr�(X;�jr;�); Mixture of Gaussians
ThenP (G = jjX = x) = PRjr=1 �jr�(X;�jr;�)�jPJ=1PR`r=1 �`r�(X;�`r;�)�`Estimate parameters by maximum likelihood of P (X;G) (possibly
subject to rank constraints!)maxrankf�jrg=K;� JXj=1Xgi=j log( RjXr=1 �jr�(xi;�jr;�)�j)
Note: reduced rank amounts to dimension reduction in predictor
space!
Stanford University—April 28, 1997 Flexible Discriminant and Mixture Models 26'
&
$
%
EM and Optimal Scoring
E-Step:
Compute memberships Prob(obs 2 rth subclass of class j jx; j)W (crjx; j) = �r�(x;�jr;�)PRjk=1 �jk�(x;�jk;�)M-Step:Construct Random Response Matrix Z with elementsW (cjrjx; j):0BBBBBBBBBB@