Nonlinear Dimensionality Reduction Approach (ISOMAP) 2006. 2. 28 Young Ki Baik Computer Vision Lab. Seoul National University.

Nonlinear Dimensionality Reduction Approach

(ISOMAP)

2006. 2. 28Young Ki Baik

Computer Vision Lab.

Seoul National University

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

References

A global geometric framework for nonlinear dimensionality reduction

J. B. Tenenbaum, V. De Silva, J. C. Langford (Science 2000)

LLE and Isomap Analysis of Spectra and Colour Images

Dejan Kulpinski (Thesis 1999)

Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering

Yoshua Bengio et.al. (TR 2003)

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Contents

Introduction

PCA and MDS

ISOMAP

Conclusion

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Dimensionality Reduction

The goalThe meaningful low-dimensional structures hidden in their high-dimensional observations.

Classical techniquesPCA (Principle Component Analysis)

– preserves the variance

MDS (MultiDimensional Scaling)

- preserves inter-point distance

ISOMAP

LLE (Locally Linear Embedding)

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Linear Dimensionality Reduction

PCA

Finds a low-dimensional embedding of the data points that best

preserves their variance as measured in the high-dimensional

input space.

MDS

Finds an embedding that preserves the inter-point distances,

equivalent to PCA when the distances are Euclidean.

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Linear Dimensionality Reduction

MDS

distances

Relation

ijd

)()( 2ji

Tjiij xxxxd

221 ijdA

matrix centering theis H , HAHB

)()( xxxxb jT

iij T

T XX(HX)(HX)Bthen

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Nonlinear Dimensionality Reduction

Many data sets contain essential nonlinear structures

that invisible to PCA and MDS

Resort to some nonlinear dimensionality reduction

approaches.

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

ISOMAP

Example of Non-linear structure (Swiss roll)

Only the geodesic distances reflect the true low-dimensional

geometry of the manifold.

ISOMAP (Isometric feature Mapping)

Preserves the intrinsic geometry of the data.

Uses the geodesic manifold distances between all pairs.

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

ISOMAP (Algorithm Description)

Step 1

Determining neighboring points within a fixed radius based on

the input space distance .

These neighborhood relations are represented as a weighted

graph G over the data points.

Step 2

Estimating the geodesic distances between all pairs of

points on the manifold by computing their shortest path

distances in the graph G.

Step 3

Constructing an embedding of the data in d-dimensional

Euclidean space Y that best preserves the manifold’s geometry.

jid ,X

jidG ,

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Step 1

Determining neighboring points within a fixed radius based on

the input space distance .

# ε-radius # K-nearest neighbors

These neighborhood relations are represented as a weighted

graph G over the data points.

ISOMAP (Algorithm Description)

jid ,X

ε

K=4

ij

k

jid ,X

kid ,X

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

ISOMAP (Algorithm Description)

Step 2

Estimating the geodesic distances between all pairs of points on the manifold by computing their shortest path distances in the graph G.

Can be done using Floyd’s algorithm or Dijkstra’s algorithm

jidG ,

)},(),( ),,(min{),(

N1,2,...,k

othewise ),(

ji, gneighborin ),(),(

jkdkidjidjid

for

jid

jidjid

GGGG

G

G

ij

k jkdG , kidG ,

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

ISOMAP (Algorithm Description)

Step 3

Constructing an embedding of the data in d-dimensional

Euclidean space Y that best preserves the manifold’s geometry.

Minimize the cost function:

)()()(

),(),(

),(

12.121

NN

GG

jiY

IDID

and

jidjiD

yyjiDwhere

2)()(LYG DDE

Solution: take top d eigenvectors of the

matrix )( GD

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Experimental results

# FACE # Hand writing

: face pose and illumination : bottom loop and top

arch

MDS : open triangles

Isomap : filled circles

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Discussion

Isomap handles non-linear manifold.

Isomap keeps the advantages of PCA and MDS.

Non-iterative procedure

Polynomial procedure

Guaranteed convergence

Isomap represents the global structure of a data set within a single coordinate system.

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU