1 M. Bronstein Multigrid multidimensional scaling Multigrid Multidimensional Scaling Michael M. Bronstein Department of Computer Science Technion – Israel.

1M. Bronstein Multigrid multidimensional scaling

Multigrid Multidimensional Scaling

Michael M. Bronstein

Department of Computer ScienceTechnion – Israel Institute of Technology


Agenda

Applications of MDS

Numerical optimization algorithms

Motivation for multiresolution MDS methods

Multigrid MDS

Experimental results

Conclusions


Dimensionality reduction

Visualization

Pattern recognition

Feature extraction

Data analysis

WRIST ROTATION

FIN

GE

R E

XT

EN

SIO

N

Low-dimensional representation of articulated hand images, showing intrinsic data dimensionality

Images: World Wide Web


Given a surface sampled at points , and the

geodesic distances on ;

Find a mapping (isometric embedding)

such that

Isometric embedding


GLOBE (HEMISPHERE) PLANAR MAP

Mapmaking

Given: geodesic distances between cities on the Earth

Find: the “best” (most distance-preserving) planar map of the cities

Optimal planar representation of the upper hemisphere of the Earth


Pattern recognition

A. Elad, R. Kimmel, Proc. CVPR 2001

ISOMETRIES OF A DEFORMABLE OBJECT

ISOMETRY-INVARIANT REPRESENTATIONS (“CANONICAL FORMS”)

Isometry-invariant representation of deformable objects using isometric embedding


Expression-invariant face recognition

ISOMETRIC EMBEDDING

FACIAL CONTOURCROPPING

FACESUBSAMPLING

CANONICAL FORM

Facial expressions ~ isometries of the facial surface

Obtain expression-invariant representation using isometric embedding

Compare the canonical forms

A. Bronstein, M. Bronstein, R. Kimmel, Proc. AVBPA 2003; IJCV 2005

Scheme of expression-invariant 3D face recognition based on isometric embedding

DISTANCESCOMPUTATION


Stress

Given a set of distances ;

and a configuration of points in -dimensional

Euclidean

space ;

Representation quality can be measured as the -distortion of the

distances (stress)


Multidimensional scaling

Stress in matrix form:

- matrix of geodesic distances (data);

- matrix of Euclidean coordinates (variable);

Multidimensional scaling (MDS) problem:

optimization variables

Optimum defined up to an isometry in


Minimization of the stress

Generic iterative optimization algorithm:

Start with an initial guess ;

At -st iteration, make a step of size in direction

such that

Repeat until a stopping condition is met, e.g.


Optimization algorithms

Gradient descent: , step size is constant orfound using line search

Newton: , step size is found using line search

Truncated Newton: direction obtained by inexact solution of

step size is chosen to guarantee descent

Quasi-Newton: direction obtained by estimating using the gradients ; step size is found using linesearch


Difficulties

Non-convex and nonlinear optimization problem (local convergence)

Hessian structured but dense

High computation complexity of and

Exact line search is prohibitive for large


SMACOF algorithm

SMACOF: steepest descent with constant step size

where

and

Can be also written as a multiplicative update

Complexity: per iteration


Multiresolution methods: motivation

Data smoothness and locality (a point can be interpolated from itsneighbors)

Complexity: - MDS problem is easier on coarser resolution

Local minima: multiple resolutions improve global convergence


Towards multigrid MDS

Convex nonlinear optimization is equivalent to a nonlinear equation

Multigrid spirit: solve problems of the form

at different resolution levels

- residual transferred from finer resolution levels

M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005


Modified stress

Problem: the function is unbounded

Modified stress:

The penalty term forces the center of mass of to zero

With modified stress, is bounded for every finite



Multigrid components

Hierarchy of grids

Restriction and prolongation operators to transfer data and

variables

from one resolution level to another

Hierarchy of optimization problems

Relaxation: steps of optimization algorithm



Coarsening schemes

In parameterization domain (suitable for parametric surfaces, e.g.acquired by 3D scanner)

Triangulation-based (suitable for general triangulated meshes)

Farthest point sampling (based on the distances matrix; suitable for arbitrary multidimensional data)


V-cycle

If (coarsest level), solve and return

Else


Relaxation

Compute

Apply MG on coarser resolution

Correction

Relaxation


Error smoothing

BEFORE RELAXATION AFTER RELAXATION

Error smoothing using SMACOF relaxation

M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear


Numerical experiments

Embedding of the “Swiss roll” surface – comparison of MDS

algorithmsconvergence in a large scale problem

Computation of canonical forms for face recognition

Sensitivity to initialization and comparison on problems of different

size

Dimensionality reduction


Experiment I: Unrolling the Swiss roll

Embedding of the Swiss roll objects into R3 using MG-MDS. N=2145

INITIALIZATION ITERATION 1 ITERATION 2 ITERATION 3 ITERATION 4



Experiment I: Convergence comparison

Convergence of different algorithms in the Swiss roll problem

COMPLEXITY (MFLOPs)

ST

RE

SS

EXECUTION TIME (sec.)



Experiment II: Facial surface embedding

Computation of a facial canonical form using MG-MDS: as few as 3 iterations are sufficient to obtain a good expression-invariant representation. N=1997

INITIALIZATION ITERATION 1 ITERATION 2 ITERATION 3



Performance of SMACOF and MG (V-cycle, 3 resolution levels) MDS algorithms using random initialization


Experiment III: Sensitivity to initialization


Boosting obtained by multigrid MDS (V-cycle, 3 resolution levels) compared to SMACOF. Initialization by the original points


Experiment III: Performance comparison


Experiment IV: Dimensionality reduction

Dimensionality reduction of 500-dimensional random data: as few as 3 iterations are sufficient to obtain distinguishable clusters.

INITIALIZATION ITERATION 1 ITERATION 2 ITERATION 3

Two sets of random binary i.i.d. 500-dimensional vectors

Set A:

Set B:


MG-MDS significantly outperforms traditional MDS algorithms(~order of magnitude)

The improvement is more pronounced for large N

MG-MDS appears to be less sensitive to initialization and has better global convergence

Conclusions


ReferencesM. M. Bronstein, A. M. Bronstein, R. Kimmel, I. Yavneh, "Multigrid multidimensional

scaling", NLAA, to appear in 2006

M. M. Bronstein, A. M. Bronstein, R. Kimmel, I. Yavneh, "A multigrid approach for multi-dimensional scaling", Proc. Copper Mountain Conf. Multigrid Methods, 2005.

A. M. Bronstein, M. M. Bronstein, and R. Kimmel. “Expression invariant face recognition:

faces as isometric surfaces”, in “Face Processing: Advanced Modeling and Methods”,Academic Press, 2005. in press.

A. M. Bronstein, M. M. Bronstein, R. Kimmel, "Three-dimensional face recognition", Intl. Journal of Computer Vision (IJCV), Vol. 64/1, pp. 5-30, August 2005.

A. M. Bronstein, M. M. Bronstein, R. Kimmel, "Expression-invariant 3D face recognition", Proc. AVBPA, Lecture Notes in Comp. Science No. 2688, Springer, pp. 62-69, 2003.

1 M. Bronstein Multigrid multidimensional scaling Multigrid Multidimensional Scaling Michael M. Bronstein Department of Computer Science Technion – Israel.

Documents

isometric embedding

distances stress slide

earth slide

line search slide

step of size

world wide web slide

gradients step size

mapping isometric