1 Manifold learning P. Agius – L13, Spring 2008 P. Agius – L13, Spring 2008 Relevant Reading: Algorithms for Manifold Learning Lawrence Cayton http://vis.lbl.gov/~romano/mlgroup/papers/manifold-learning.pdf Handling the curse of dimensionality via dimensionality reduction High dimensional data is often simpler than it’s high dimension suggests. Want: simplified non-overlapping representation of the data with features identifiable with the underlying patterns of the data in its original format … want to discover a manifold structure in the data. Most popular: PCA finds directions of max variance and finds basis for a linear subspace. But only appropriate if data lies in linear subspace. Manifold algorithms are non-linear analogs to PCA
31
Embed
Manifold learning - ut · Manifold learning …properly defined Some algorithms: Isomap Locally Linear Embedding LaplacianEigenmaps SemidefiniteEmbedding Parameter for k-nearest neighbors
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Manifold learning
P. Agius – L13, Spring 2008
P. Agius – L13, Spring 2008
Relevant Reading: Algorithms for Manifold Learning
Handling the curse of dimensionality via dimensionality reduction
High dimensional data is often simpler than it’s high dimension suggests.
Want: simplified non-overlapping representation of the data with features identifiable with the underlying patterns of the data in its original format … want to discover a manifold structure in the data.
Most popular: PCA finds directions of max variance and finds basis for a linear subspace. But only appropriate if data lies in linear subspace.
Manifold algorithms are non-linear analogs to PCA
2
P. Agius – L13, Spring 2008
High dimensional inputs: X =
Low dimensional outputs: Y=
n=number of points, D=number of input dimensions, d=number of manifold dimensions
P. Agius – L13, Spring 2008
From 3 dimensions to 1 dimension …
this 3-dim curve can be represented as a line in 1 dimension
In topology … definition of a homeomorphism is: a continuous function whose inverse is also continuous.
If we can find such a function from high-dim space to low-dim space, then we can say they are homeomorphic.
3
P. Agius – L13, Spring 2008
Manifold learning … properly defined
Some algorithms:
IsomapLocally Linear EmbeddingLaplacian EigenmapsSemidefinite Embedding
Parameter for
k-nearest neighbors
P. Agius – L13, Spring 2008
Isomap – isometric feature mapping
Reminiscent of MDS … indeed it is an extension of MDS
Two main steps:
- Estimate the geodesic distances (i.e. distances in manifold)
- Use MDS to find points in low-dim Euclidean space
4
P. Agius – L13, Spring 2008
Estimating geodesic distances
Two assumptions for Isomap: -there is a mapping that preserves distances.- manifold is smooth enough so that distances are approximately linear
For points that are far apart in the manifold, linear approx is no good. For such points:
- build a k-nearest neighbor graph weighted by Euclidean distances between points
- then find distance between the far points using a shortest path algorithm (Eg. Dijkstra)
P. Agius – L13, Spring 2008
Isomap uses MDS to find those points whose geodesic distances match the Euclidean distances.
5
P. Agius – L13, Spring 2008
Locally Linear Embedding (LLE)
Intuition: visualize manifold as collection of overlapping coordinate patches. If neighborhoods are sufficiently small and manifold sufficiently smooth, then patches are approx linear.
Goal: identify these linear patches, characterize their geometry and find a mapping accordingly
P. Agius – L13, Spring 2008
6
P. Agius – L13, Spring 2008
Laplacian Eigenmaps
Based on spectral graph theory …
Given a graph with weights W, define the graph Laplacian as
L=D-W
where D=diagonal matrix with
Eigenvalues and eigenvectors of L reveal lots of info about graph
P. Agius – L13, Spring 2008
k-nearest neighbors of xj
7
P. Agius – L13, Spring 2008
Semidefinite Embedding
Intuition: Imagine k-nearest neighbors of each point to be connected by a rigid rod. Now take this structure and pull apartas far as possible … can we properly unravel it???
P. Agius – L13, Spring 2008
Constraint that ensures
distances between neighbor
points are preserved
B=psd matrix
8
P. Agius – L13, Spring 2008
Other algorithms
Hessian LLE (hLLE) – replace graph Laplacian with a Hessian estimator
Local Tangent Space alignment (LTSA) – similar to hLLE, estimates the tangent space at each point by performing PCA
Paper discusses a total of 6 algorithms … and there are variations … so many! Why so many?
Evaluating these algorithms is difficult. Choosing the best is also difficult.
P. Agius – L13, Spring 2008
Comparisons
Isometric embedding (assumption of distance preservation)– Isomap, hLLE, Semidefinite Embedding
versusConformal embedding (preservation of angles)
- c-Isomap
Local versus Global- Isomap is global, all point pairs considered during embedding- LLE is local, cost function only considers k-nearest neighbors
Time complexity – spectral decomposition is costly!
9
P. Agius – L13, Spring 2008
Open questions
Other issue not much addressed: choice of k in k-nearest neighbors approaches