Out-of-Sample Extension and Reconstruction on Manifolds

OUT-OF-SAMPLE EXTENSION AND RECONSTRUCTION ON MANIFOLDSBhuwan DhingraFinal Year (Dual Degree)Dept of Electrical Engg.

INTRODUCTION

An m-dimensional manifold is a topological space which is locally homeomorphic to the m-dimensional Euclidean space

In this work we consider manifolds which are: Differentiable Embedded in a Euclidean space Generated from a set of m latent variables via a

smooth function f

INTRODUCTION

n >> m

NON-LINEAR DIMENSIONALITY REDUCTION

In practice we only have a sampling on the manifold

Y is estimated using a Non-Linear Dimensionality Reduction (NLDR) method

Examples of NLDR methods –ISOMAP, LLE, KPCA etc.

However most non-linear methods only provide the embedding Y and not the mappings f and g

PROBLEM STATEMENT

x*y*

g

f

OUTLINE

p is the nearest neighbor of x* Only the points in are used for extension and

reconstruction

OUTLINE

The tangent plane is estimated from the k-nearest neighbors of p using PCA

𝑥𝑖∈𝑁𝑝𝑘

OUT-OF-SAMPLE EXTENSION

A linear transformation Ae is learnt s.t Y = AeZ

Embedding for new point y* = Aez*

�̂�𝑝∈𝑍 𝑦 𝑝∈𝑌Ae

z* y*

OUT-OF-SAMPLE RECONSTRUCTION

A linear transformation Ar is learnt s.t Z = ArY

Projection of reconstruction on tangent plane z* = Ary*

�̂�𝑝∈𝑍 𝑦 𝑝∈𝑌

z* y*Ar

PRINCIPAL COMPONENTS ANALYSIS

Covariance matrix of neighborhood:

Let be the eigenvector and eigenvalue matrixes of Mk

Then

Denote then the projection of a point x onto the tangent plane is given by:

LINEAR TRANSFORMATION

Y and Z are both centered around and Then Ae =BeRe where Be and Re are scale and

rotation matrices respectively If is the singular value decomposition

of ZTY, then

FINAL ESTIMATES

ERROR ANALYSIS

We don’t know the true form of f or g to compare our estimates against

Reconstruction Error: For a new point x* its reconstruction is computed as , and the reconstruction error is

SAMPLING DENSITY

To show: As the sampling density of points on the manifold increases, reconstruction error of a new point goes to 0

In a k-NN framework, the sampling density can increase in two ways: k remains fixed and the sampling width

decereases remains fixed and

We consider the second case

NEIGHBORHOOD PARAMETERIZATION

Assume that the first m-canonical vectors of are along

RECONSTRUCTION ERROR

But ArAe = I, hence

RECONSTRUCTION ERROR

Tyagi, Vural and Frossard (2012) derive conditions on k s.t the angle between and is bounded

They show that as

Equivalently, where Rm is an aribitrary m-dimensional rotation matrix

and

RECONSTRUCTION ERROR

Hence the reconstruction approaches the projection of x* onto

SMOOTHNESS OF MANIFOLD

If the manifold is smooth then all will be smooth

Taylor series of :

As because x* will move closer to p

RESULTS - EXTENSION

Out of sample extension on the Swiss-Roll dataset

Neighborhood size = 10

RESULTS - EXTENSION

Out of sample extension on the Japanese flag dataset

Neighborhood size = 10

RESULTS - RECONSTRUCTION

Reconstructions of ISOMAP faces dataset (698 images)

n = 4096, m = 3 Neighborhood size = 8

RECONSTRUCTION ERROR V NUMBER OF POINTS ON MANIFOLD

ISOMAP Faces dataset Number of cross validation sets = 5 Neighborhood size = [6, 7, 8, 9]