Classification Supervised and unsupervised

Classification Supervised and unsupervised

Tormod Næs

Matforsk

and

University of Oslo

Classificaton

• Unsupervised (cluster analysis)– Searching for groups in the data

• Suspicion or general exploration– Hierarchical methods, partitioning methods

• Supervised (discriminant analysis)– Groups determined by other information

• External or from a cluster analysis

– Understand differences between groups

– Allocate new objects to the groups• Scoring, finding degree of membership

Group 1

Group 2

New object X

?

?

What is the difference? Where?

Why supervised classification?

• Authenticity studies– Adulteration, impurities, different origin, species

etc.• Raw materials• Consumer products according to specification

• When quality classes are more important than chemical values

• raw materials acceptable or not• raw materials for different products

Flow chart for discriminant analysis

Main problems

• Selectivity– Multivariate methods are needed

• Collinearity– Data compression is needed

• Complex group structures– Ellipses, squares or ”bananas”?

X1

X2

Authentic

Adulterated

The selectivity problem

Solving the selectivity problem

• Using several measurements at the same time– The information is there!

• Multivariate methods. These methods combine several instrumental NIR variables in order to determine the property of interest

• Mathematical ”purification” instead of wet chemical analysis

Multivariate methodsToo many variables can also sometimes create

problems

– Interpretation– Computations, time and numerical stability– Simple and difficult regions (nonlinearity)– Overfitting is easier (dependentent on method used)

• Sometimes important to find good compromises (variable selection)

Conflict between flexibility and stability

Estimation error

Model error

Some main classes of methods

• Classical Bayes classification– LDA, QDA

• Variants, modifications used to solve the collinearity problem– RDA, DASCO, SIMCA

• Classification based on regression analysis– DPLS, DPCR

• KNN methods, flexible with respect to shape of the groups

Bayes classification

• Assume prior probabilities pj for the groups

– If unknown, fix them to be pj= 1/C or

– equal to the proportions in the dataset

• Assume known probability model within each class (fj(x))

– Estimated from the data, usually covariance matrices and means

Bayes classification• +

• well understood, much used, often good properties, easy to validate

• easy to modify for collinear data

• Easy to updated, covariances

• Can be modified for cost

• Outlier diagnostics (not directly, but can be done, M-distance)

• - • Can not handle too complex group structures, designed for elliptic

structures

• not so easy to interpret directly

• often followed by a Fisher’s linear discriminant analysis. Directly related to interpreting differences between groups

Bayes ruleMaximise porterior probability

Normal data, minimise

Estimate model parameters,

jjjijT

jii xxL log2log)()( 1

jjjijT

jii xxL log2ˆlog)ˆ(ˆ)ˆ(ˆ 1

Mahalanobis distance plus determinant minus prior probability

Different covariancestructures

Mahalanobis distance is constant on ellipsoids

Best known members

• Equal covariance matrix for each group– LDA

• Unequal covariance matrices– QDA

• Collinear data, unstable inverted covariance matrix (see equation)– Use principal components (or PLS components)

– RDA, DASCO estimate stable inverse covariance matrices

Classification by regression

• 0,1 dummy variables for each group• Run PLS-2 (or PCR) or any other method which solves the

collinearity• Predict class membership.

– The class with the highest value gets the vote• All regular interpretation tools are available, variable selection,

plotting outliers diagnostics etc.• Linear borders between subgroups, not too complicated groups.• Related to LDA, not covered here• If large data sets, we can use more flexible methods

Example, classification of mayonnaise based on different oils

The oils were•soybean•sunflower•canola•olive•corn•grapeseed

Indahl et al (1999). Chemolab

16 samples in each group

, Feasibility study, authenticity

Classification properties of QDA, LDA and regression

Start out low

Comparison

• LDA and QDA gave almost identical results

• It was substantially better to use LDA/QDA based on PLS/PCA components instead of using PLS directly

Fisher’s linear discriminant analysis

• Closely related to LDA

• Focuses on interpretation

– Use “spectral loadings” or group averages

• Finds the directions in space which distinguish the most between groups

– Uncorrelated

• Sensitive to overfitting, use PC’s first

Fisher’s method. Næs, Isaksson, Fearn and Davies (2001). A user friendly guide to cal. and class.

Not possible to distinguish the groups from each other

Plot of PC1 vs PC2

Mayonnaise data, clear separation

Canonical variates based on PC’s

PCA Fisher’s method

Forina et al(1986), Vitis

Italian wines from same region, but based on different cultivars,27 chromatic and chemical variables

Barolo

Grignolino

Barbera

Error ratesValidated properly

• LDA – Barolo 100%, Grignolino 97.7%, Barbera

100%

• QDA– Barolo 100%, Grignolino 100%, Barbera100%

KNN methods

• No model assumptions

• Therefore: needs data from “everywhere” and many data points

• Flexible, complex data structures

• Sensitive to overfitting, use PC’s

New sample

KNN, finds the N samples which are closestIn this case 3 samples

Cluster analysisUnsupervised classification

• Identifying groups in the data

– Explorative

Examples of use

• Forina et al(1982). Olive oil from different regions (fatty acid composition). Ann. Chim.

• Armanino et al(1989), Olive oils from different Tuscan provinces (acids, sterols, alcohols). Chemolab.

Methods

• PCA (informal/graphical)– Look for structures in scores plots

– Interpretation of subgroups using loadings plots

• Hierarchical methods (more formal)– Based on distances between objects (Euclidean or

Mahalanobis)

– Join the two most similar

– Interpret dendrograms

Armanino et al(1989), Chem.Int. lab. Systems.

120 olive oils from one region in Italy, 29 variables (fatty acids, sterols, etc.)