This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
• A classifier that places a pattern in one of two classes is often referred to as a dichotomizer.
• We can reshape the decision rule:
02121 )() - g(g) g() (g)(if g xxxxx• If we use log of the posterior probabilities:
)ln()ln()ln(
)()(
2
1
2
1
21
PP
pp
)g()f(
PP)g(
xx
xx
xxx
• A dichotomizer can be viewed as a machine that computes a single discriminant function and classifies x according to the sign (e.g., support vector machines).
Two-Category Case (Review)
ECE 8443: Lecture 04, Slide 3
Unconstrained or “Full” Covariance (Review)
ECE 8443: Lecture 04, Slide 4
• This has a simple geometric interpretation:
)()(
ln2 222
i
jji P
P
xx
• The decision region when the priors are equal and the support regions are spherical is simply halfway between the means (Euclidean distance).
• Bayes decision rule guarantees lowest average error rate• Closed-form solution for two-class Gaussian distributions• Full calculation for high dimensional space difficult• Bounds provide a way to get insight into a problem and
engineer better solutions.
• Need the following inequality:100,],min[ 1 andbababa
Assume a b without loss of generality: min[a,b] = b.
Also, ab(1- ) = (a/b)b and (a/b) 1.
Therefore, b (a/b)b, which implies min[a,b] ab(1- ) .
• Apply to our standard expression for P(error).
Error Bounds
ECE 8443: Lecture 04, Slide 11
xxx
xxx
xxx
xxx
xx
xxxxx
dppPP
dpPpP
dpPpP
dpp
pPp
pP
dpPPerrorP
)()()()(
)()()()(
)]()(),()(min[
)(])(
)()(,
)()()(
min[
)()](),(min[)(
21
121
1
21
21
11
2211
2211
21
• Recall:
• Note that this integral is over the entire feature space, not the decision regions (which makes it simpler).
• If the conditional probabilities are normal, this expression can be simplified.
Chernoff Bound
ECE 8443: Lecture 04, Slide 12
• If the conditional probabilities are normal, our bound can be evaluated analytically:
))(exp()()( 21
1 kdpp xxx
where:
)1(21
21
121
2112
)1(ln
21
)(])1([)(2
)1()(
tk
• Procedure: find the value of that minimizes exp(-k( ), and then compute P(error) using the bound.
• Benefit: one-dimensional optimization using
Chernoff Bound for Normal Densities
ECE 8443: Lecture 04, Slide 13
• The Chernoff bound is loose for extreme values• The Bhattacharyya bound can be derived by = 0.5:
))(exp()()(
)()()()(
)()()()(
21
2121
21
121
1
kPP
dppPP
dppPP
xxx
xxx
where:
21
21
12121
122ln
21)(]
2[)(
81)(
tk
• These bounds can still be used if the distributions are not Gaussian (why? hint: Occam’s Razor). However, they might not be adequately tight.