Fast N-Body Learning Nando de Freitas University of British Columbia.

Fast N-Body LearningFast N-Body Learning

Nando de Freitas

University of British Columbia

Historical Perspective Historical Perspective

• Non-iterative or “direct” methods for eigenvalue problems and linear systems of equations require O(N3) operations.

• Let's look at the history of what has been regarded as large N:

1950: N=20 1965: N=200 1980: N=2000 1995: N=20000

• So over the course of 45 years N has increased by a factor of103. However, the speed of computers has increased by a factor of109. From this the O(N3) bottleneck is evident.

If only we could reduce the cost to O(N) – sigh!

Krylov for Eigen-Problems Krylov for Eigen-Problems

Krylov for Systems of Equations Krylov for Systems of Equations

N-Body Problems in Learning N-Body Problems in Learning

Sum-kernel problem:

Max-kernel problem:

N-Body Problems N-Body Problems

Obvious applications of N-body Learning Obvious applications of N-body Learning

• Exact and approximate message propagation.

• Markov chain Monte Carlo

• Gaussian processes, Wishart processes and Laplace processes.

• Spectral learning: eigenmaps, SNE, NCUTS, ranking on manifolds, … (even if using Nystrom)

• Reinforcement learning.

• The E step.

• Kernel-(fill in your favourite name).

• Rao-Blackwellised Monte Carlo.

• Nearest neighbour methods.

• Some types of boosting.

• Computer graphics.

• EM, fluid dynamics, gravitation, quantum systems.

• … and much more !

Obvious applications of N-body Learning Obvious applications of N-body Learning

Illustrative Example: Zhu, Lafferty & Zoubin Illustrative Example: Zhu, Lafferty & Zoubin

Illustrative Example Illustrative Example

Energy function using the graph Laplacian:

Easy, but … a big linear system:

Naïve iterative solution:

Illustrative Example Illustrative Example

We have solved a Gaussian process (where the covariance is the inverse graph Laplacian in O(N).

Illustrative Example Illustrative Example

Message propagation Message propagation

Whether it’s exact:

… or approximate:

Fast Methods in this Workshop Fast Methods in this Workshop

Fast Multipole Methods Fast Multipole Methods

Recursive Tree Structures Recursive Tree Structures

Recursive Tree Structures Recursive Tree Structures

Distance Transform Distance Transform

m(j) = min ( w(i) + d(i,j) )i

In this workshop In this workshop

•You’ll encounter tutorials on fast methods from the people who’ve been developing them.

• You’re likely to see encounter people arguing over error bounds, implementation strategies, applications and many more things.

• You’ll see statistics, learning, data structures and numerical computation come together.

• You’ll dream of the powder up on the hill.