Pattern Classification & Decision Theory. How are we doing on the pass sequence? Bayesian regression and estimation enables us to track the man in the.

Pattern Classification & Decision Theory

How are we doing on the pass sequence?• Bayesian regression and estimation enables us to

track the man in the striped shirt based on labeled data• Can we track the man in the white shirt?

Not very well.

Feature, xHan

d-l

abe

led

ho

rizo

nta

l co

ord

inat

e, t

Regression fails to identify that there really are two classes of solution

Decision theory

• We wish to classify an input x as belonging to one of K classes, where class k is denoted

• Example: Buffalo digits, 10 classes, 16x16 images, each image x is a 256-dimensional vector, xm [0,1]

Decision theory

• We wish to classify an input x as belonging to one of K classes, where class k is denoted

• Partition the input space into regions , , …, so that if x , our classifier predicts class

• How should we choose the partition?

• Suppose x is presented with probability p(x) and the

distribution over the class labels given x is p( |x)

• Then, p(correct) = k x p(x) p( |x) dx

• This is maximized by assigning each x to the region

whose class maximizes p( |x)

Three approaches to pattern classification

1. Discriminative and non-probabilistic– Learn a discriminant function f (x),

which maps x directly to a class label

2. Discriminative and probabilistic– For each class k, learn the probability

model

– Use this probability to classify a new input x

2

1

0

3

f(x)

Three approaches to pattern classification

3. Generative– For each class k, learn the generative

probability model– Also, learn the class probabilities– Use Bayes’ rule to classify a new

input:

where

Three approaches to pattern classification

1. Discriminative and non-probabilistic– Learn a discriminant function f (x),

which maps x directly to a class label

2. Discriminative and probabilistic– For each class k, learn the probability

model

– Use this probability to classify a new input x

2

1

0

3

f(x)

Can we use regression to learn discriminant functions?

2

1

0

3

f(x)

2

1

0

3

f(x)2

1

0

3

f(x)

• What do the classification regions look like?• Is there any sense in which square error is an

appropriate cost function for classification?• We should be careful to not interpret integer-valued

class labels as ordinal targets for regression

Can we use regression to learn discriminant functions?

The one-of-K representation

• For > 2 classes, each class is represented by a binary vector t with 1 indicating the class:

• K regression problems:

• To classify x, pick class k with largest yk(x)

100...

0

. . .t =

010...

0

t =

Let’s focus on binary classification

• Predict target t {0,1} from input vector x

• Denote the mth input of training case n by xnm

Classification boundary for linear regression

• Values of x where y(x)=0.5 are ambiguous – these form the classification boundary

• For these points, = 0.5

• If x is (M+1)-dimensional, this defines an M-dimensional hyperplane separating the classes

How well does linear regression work?

Works well in some cases, but there are two problems:

• Due to linearity, “extreme” x’s cause extreme y(x,w)’s• Due to squared error, extreme y(x,w)’s dominate learning

Logistic regression(more later)

Linear regression

Clipping off extreme y’s

• To clip off extremes of , we can use a sigmoid function:

where

• Now, squared error won’t penalize extreme x’s

• y is now a non-linear function of w, so learning is harder

How the observed error propagates back to the parameters

E(w) = ½ n ( tn – (mwmxnm) )2

• The rate of change of E w.r.t. wm is

E(w)/wm = -n ( tn - yn ) yn (1- yn) xnm

– Useful fact: ’(a) = (a) (1 - (a)),

• Compare with linear regression:

E(w)/wm = -n ( tn - yn ) xnm

yn

The effect of clipping

• Regression with sigmoid:

E(w)/wm = -n ( tn - yn ) yn (1- yn) xnm

• Linear regression:

E(w)/wm = -n ( tn - yn ) xnm

For these outliers, both (tn-yn) 0 and y(1-y) 0, so the outliers won’t hold back improvement of the boundary

Squared error for learning probabilities

• If t = 0 and y 1, y is moderately pulled down (grad

1)

• If t = 0 and y 0, y is weakly pulled down (grad 0)

E = ½(t-y)2

y

t = 0E dE/dy = 1

dE/dy = 0

Problems:• Being certainly wrong is often undesirable• Often, tiny differences between small probabilities count a lot

Three approaches to pattern classification

1. Discriminative and non-probabilistic– Learn a discriminant function f (x),

which maps x directly to a class label

2. Discriminative and probabilistic– For each class k, learn the probability

model

– Use this probability to classify a new input x

2

1

0

3

f(x)

Logistic regression: Binary likelihood

• As before, we use:

where

• Now, use binary likelihood: p(t|x) = y(x)t(1-y(x))1-t

• Data log-likelihood:

L = n tn ln(mwmxnm) + (1-tn) ln(1-(mwmxnm))

• Unlike linear regression, L is nonlinear in the

w’s, so gradient-based optimizers are needed

Binary likelihood for learning probabilities

• If t = 0 and y 1, y is strongly pulled down (grad ∞)

• If t = 0 and y 0, y is moderately pulled down (grad

1)

E = -ln(1-y)

y

t = 0E

dE/dy = 1

dE/dy = 0

E = ½(t-y)2

t = 0dE/dy = 1

dE/dy ∞

How the observed error propagates back to the parameters

L = n tn ln(mwmxnm) + (1-tn) ln(1-(mwmxnm))

• The rate of change of L w.r.t. wm is

L/wm = n ( tn - yn ) xnm

• Compare with sigmoid plus squared error:

E(w)/wm = -n ( tn - yn ) yn (1- yn) xnm

• Compare with linear regression:

E(w)/wm = -n ( tn - yn ) xnm

yn

How well does logistic regression work?

Logistic regression

Linear regression

Multiclass logistic regression

• Create one set of weights per class and define

• The K-class generalization of the sigmoid function is

p(t|x) = k exp(tk yk(x)) / k exp(yk(x))

which is equivalent to

p( |x) = exp(yk(x)) / j exp(yj(x))

• Learning: Similar to logistic regression (see textbook)

Three approaches to pattern classification

3. Generative– For each class k, learn the generative

probability model– Also, learn the class probabilities– Use Bayes’ rule to classify a new

input:

where

Gaussian generative models

• We can assume each element of x is independent and Gaussian, given the class:

p(x| ) = m p(xm| ) = m N (xm | km,km2)

• Contour plot of density:

k1

k2

k1

k2

Isotropic Gaussian:k1

2 = k22

Learning a Buffalo digit classifier(5000 training cases)

• The generative ML estimates of km and km2 are just

the data means and variances:

Means:

Variances (black=low variance, white=high variance):

• The classes are equally frequent, so = 1/10

• To classify a new input x, compute (in the log-domain!)

A problem with the ML estimate

• Some pixels were constant across all training images within a class, so ML

2 = 0

• This causes numerical problems when evaluating Gaussian densities, but is also an overfitting problem

• Common hack: Add min2 to all variances

• More principled approaches– Regularize 2

– Place a prior on 2 and use MAP

– Place a prior on 2 and use Bayesian learning

Classifying test data(5000 test cases)

• Adding min2 = 0.01 to the variances, we obtain:

– Error rate on training set = 16.00% (std dev .5%)

– Error rate on test set = 16.72% (std dev .5%)

• Effect of value of min2 on error rates:

log10 min2

Test error rate

Training error rate

Full-covariance Gaussian models

• Let x = y, where y is isotropic Gaussian and is an

M x M rotation and scale matrix

• This generates a full-covariance Gaussian:

• Defining = (-1T-1)-1, we obtain

where is the covariance matrix: jk = COV(xj, xk)Determinant

Generative models easily induce non-linear decision boundaries

• The following three-class problem shows how three axis-aligned Gaussians can induce nonlinear decision boundaries

Generative models easily induce non-linear decision boundaries

• Two Gaussians can be used to account for “inliers” and “outliers”

How are we doing on the pass sequence?• Bayesian regression and estimation enables us to

track the man in the striped shirt based on labeled data• Can we track the man in the white shirt?

Not very well.

Feature, xHan

d-l

abe

led

ho

rizo

nta

l co

ord

inat

e, t

Regression fails to identify that there really are two classes of solution

Po

siti

on

of

man

in

wh

ite

shir

t

Using classification to improve trackingThe position of the man in the striped shirt can be used to

classify the tracking mode of the man in the white shirt

Feature

Po

siti

on

of

man

in

str

iped

sh

irt

Feature

Po

siti

on

of

man

in

wh

ite

shir

t, tw

Using classification to improve tracking• xs and ts = feature and position of man in striped shirt

• xw and tw = feature and position of man in white shirt

• For man in white shirt, hand-label regions and and learn two trackers

Feature, xs

Po

siti

on

of

man

in

str

iped

sh

irt,

ts

Feature, xw

p( |ts)

p(tw|xw, )p(ts|xs) p( |ts)

p(tw|xw, )

Using classification to improve tracking

• The classifier can be obtained using the generative approach, where each class-conditional likelihood is a Gaussian

• Note: Linear classifiers won’t work

Feature, xs

Po

siti

on

of

man

in

str

iped

sh

irt,

ts

p( |ts)

p(ts|xs) p( |ts)

p(ts| )

p(ts| )

Questions?

How are we doing on the pass sequence?

• We can now track both men, provided with– Hand-labeled coordinates of both men in 30 frames– Hand-extracted features (stripe detector, white blob

detector)– Hand-labeled classes for the white-shirt tracker

• We have a framework for how to optimally make decisions and track the men

How are we doing on the pass sequence?

• We can now track both men, provided with– Hand-labeled coordinates of both men in 30 frames– Hand-extracted features (stripe detector, white blob

detector)– Hand-labeled classes for the white-shirt tracker

• We have a framework for how to optimally make decisions and track the men

This takes too much time to do by hand!

Lecture 4 Appendix

Binary classification regions for linear regression

• is defined by , and vice versa for

• Values of x satisfying are on the decision boundary, which is a D-1 dimensional hyperplane– w specifies the orientation

of the decision hyperplane

– -w0/||w|| specifies the

distance from the hyperplane

to the origin

– The distance from input xto the hyperplane is

y(x)/||w||

K-ary classification regions for linear regression

• x if

• Each resulting classification region is contiguous and has linear boundaries:

Fisher’s linear discriminant and least squares• Fisher: Viewing y = wTx as a projection, pick w to maximize

the distance between the means of the data sets, while also minimizing the variances of the data sets

• This result is also obtained using linear regression, by setting t = N/N1 for class 1 and t = - N/N2 for class 2, where Nk = #

training cases in class k

In what sense is logistic regression linear?

• The log-odds can be written thus:

• Each input contributes linearly to the log-odds

1-

Gaussian likelihoods and logistic regression• For two classes, if their covariance matrices are equal,

1=2=, we can write the log-odds as

• So…

=

2 2 1 1=

=

where

Logistic regression classifiers

Classifiers using equal-covariance Gaussian generative models

“Linear models” or classifiers with “linear boundaries”

Don’t be fooled• Such classifiers can be very hard to learn• Such classifiers may have boundaries that are highly

nonlinear in x (eg, via basis functions)

• All this means is that in some space the boundaries are linear