Physics, Mathematics and Computers - Fernando Pérezfperez.org/talks/0705_berkeley.pdfPhysics/Math MRA IPython Parallel Other projects Physics, Mathematics and Computers Better algorithms

Physics/Math MRA IPython Parallel Other projects

Physics, Mathematics and ComputersBetter algorithms + Better tools = Better science!

Fernando Pérez

Department of Applied MathematicsUniversity of Colorado, Boulder.

U. C. BerkeleyMay 16, 2007


Outline

1 Physics and Mathematics

2 Numerical Multiresolution Algorithms

3 Computing with Python and IPython

4 The future: IPython and parallel computing

5 Other projects of interest

F. P. (App. Math - CU Boulder) Physics, Mathematics and Computers Berkeley 2 / 45


Outline








Outline








Outline








Outline








Outline








My background: theoretical physics, numerics

Physics

Lattice QCD (Quantum Chromo Dynamics): quarks and gluons.Classical chaos: classical 3-body Coulomb system.Quantum chaos: a quantum version of Lyapunov exponents.

Applied Mathematics

Algorithm design for computational physics.Strong connection between the physics and the numerics.

Computing

Tools I picked up as I needed them.I ended up writing many of them, because they didn’t exist...or what existed was not good enough.




Physics


Applied Mathematics


Computing





Physics


Applied Mathematics


Computing




Why am I here?

Evolution of intentParticle physics, quantum chaos: extremely theoretical.

Applied Mathematics: ideas that are actually applicable.

Computing: shattered my prejudice against practicality.

Consistency of background

Physics: a solid understanding of nature’s basic mechanisms.

Math: robust approaches to extract valid, reliable answers.

Computing: build practical tools that support the scientific effort.

Why neuroscience?

The life sciences are now quantitiative.

A fascinating multiscale problem.

Very well suited to my background (renormalization, wavelets,...)



Why am I here?








Why neuroscience?






Why am I here?








Why neuroscience?






Some physical problems of interest

Their natural formulation is a (Partial) Differential Equation (PDE):

Poisson: gravity, electricity, ...

−∇2u = f

Schrödinger: quantum mechanics(−1

2∇

2 +V

)ψ = Eψ

Navier-Stokes (modified form): fluids

αv−µ∇2v+∇p = f

∇ ·v = 0

A good fraction of the world’s (scientific) computing time is spent on these.



What are we after?

Immediate Goals

Accurate and reliable methods of solution...

that work efficiently (time and storage)...

in more than one dimension...

and for a wide class of problems.

Overall program

A robust formulation: apply integral operators instead of invertingdifferential ones (don’t divide by zero!).

Build approximations with finite but controlled precision.

Develop multiscale, fully adaptive algorithms...

That generalize naturally to more than one dimension.



What are we after?

Immediate Goals

Accurate and reliable methods of solution...

that work efficiently (time and storage)...

in more than one dimension...

and for a wide class of problems.

Overall program

A robust formulation: apply integral operators instead of invertingdifferential ones (don’t divide by zero!).

Build approximations with finite but controlled precision.

Develop multiscale, fully adaptive algorithms...

That generalize naturally to more than one dimension.



Why do we need fast algorithms?

Because computers are getting bigger and faster!

How do you get a fast algorithm?

Compress: factorize, find an alternate representation, approximate, decouple.







































Multiresolution algorithms in multiple dimensions

Key mathematical ideas:1 Multiresolution analysis (wavelets): sparse representations (compress).2 Separated representations: reduction of dimensionality cost (approximate,

decouple).

Group effort over many years (1988-today):1 Gregory Beylkin, Lucas Monzón, Christopher Kurcz - CU Boulder2 Martin Mohlenkamp - Ohio University3 Robert Harrison, George Fann, Takeshi Yanai, Zhengting Gan - ORNL4 Vani Cheruvu - (now at NCAR)5 Robert Cramer - (now at Raytheon)



Multiresolution algorithms in multiple dimensions

Key mathematical ideas:1 Multiresolution analysis (wavelets): sparse representations (compress).2 Separated representations: reduction of dimensionality cost (approximate,

decouple).

Group effort over many years (1988-today):1 Gregory Beylkin, Lucas Monzón, Christopher Kurcz - CU Boulder2 Martin Mohlenkamp - Ohio University3 Robert Harrison, George Fann, Takeshi Yanai, Zhengting Gan - ORNL4 Vani Cheruvu - (now at NCAR)5 Robert Cramer - (now at Raytheon)


Physics/Math MRA IPython Parallel Other projects Functions Operators d = 1 recap

Outline








A specific example: Poisson’s equation

The solution to∇

2φ(r) = ρ(r)

can be written in integral form as

φ(r) =Z

G(r, r′)ρ(r′)d3r′ =Z

1|r− r′|

ρ(r′)d3r′.

We need:

A way of representing ρ(r′) and G(r, r′) efficiently (very different problems)

A way of applying the integral (a generalized multiplication).

This is a full 3-dimensional problem.

We’d like a method that easily generalizes to other G(r, r′) functions.



A specific example: Poisson’s equation

The solution to∇

2φ(r) = ρ(r)

can be written in integral form as

φ(r) =Z

G(r, r′)ρ(r′)d3r′ =Z

1|r− r′|

ρ(r′)d3r′.

We need:

A way of representing ρ(r′) and G(r, r′) efficiently (very different problems)

A way of applying the integral (a generalized multiplication).

This is a full 3-dimensional problem.

We’d like a method that easily generalizes to other G(r, r′) functions.



Multiresolution analysis, intuitively

Imagine a simple signal f (t) you want to study:

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 0.2 0.4 0.6 0.8 1

At each scale n, divide the unit interval [0,1] into 2n binary subintervals:



Multiresolution analysis, intuitively

Imagine a simple signal f (t) you want to study:

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 0.2 0.4 0.6 0.8 1

At each scale n, divide the unit interval [0,1] into 2n binary subintervals:


Multiresolution basics (2)

And compute:

Average values of function at level n : space Vn.

Differences between successive levels: space Wn = Vn+1 −Vn.

f (t) can studied (compressed, denoised, ...) from {V0,W0,W1, . . .} :

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 0.2 0.4 0.6 0.8 1

V2V4

V9-full

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0 0.2 0.4 0.6 0.8 1

W2W4W8

Note: the coefficients in Wn are small and localized around changes.

Multiresolution basics (2)

And compute:

Average values of function at level n : space Vn.

Differences between successive levels: space Wn = Vn+1 −Vn.

f (t) can studied (compressed, denoised, ...) from {V0,W0,W1, . . .} :

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 0.2 0.4 0.6 0.8 1

V2V4

V9-full

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0 0.2 0.4 0.6 0.8 1

W2W4W8

Note: the coefficients in Wn are small and localized around changes.

Functions: adaptive, controlled accuracy decompositions

Nnod = 12, ǫ = 1.0e− 10, Nblocks = 21

Nnod = 10, ǫ = 5.0e− 11, Nblocks = 634

Functions: adaptive, controlled accuracy decompositions

Nnod = 12, ǫ = 1.0e− 10, Nblocks = 21

Nnod = 10, ǫ = 5.0e− 11, Nblocks = 634

Sparse operatorsSparse: having few non-zero entries (compressible → fast→ good!).

Instead of applying this:

We apply this (blue is zero, not done):

Sparse operatorsSparse: having few non-zero entries (compressible → fast→ good!).

Instead of applying this:

We apply this (blue is zero, not done):


Adaptive natural-scale applicationA graphical representation



For one dimension (d = 1), lessons learned

The Good1 Multiwavelets → sparse operators → fast algorithms.2 Accuracy is guaranteed by construction.3 We efficiently handle multi-scale interactions.4 Hierarchical, adaptive algorithm.

The BadThis approach does not directly extend to d > 1 .



For one dimension (d = 1), lessons learned

The Good1 Multiwavelets → sparse operators → fast algorithms.2 Accuracy is guaranteed by construction.3 We efficiently handle multi-scale interactions.4 Hierarchical, adaptive algorithm.

The BadThis approach does not directly extend to d > 1 .


Operators(d > 1): Gaussians to the rescueUse Gaussians for approximations

1‖r− r′‖

≈M

∑m=1

wme−τm‖r−r′‖2

,

with controlled accuracy ε:

The problem factorizes partially: a practical solution for d > 1!

φijk =N

∑i ′ j ′k ′=1

Gii ′,jj ′,kk ′ρi ′ j ′k ′ =M

∑m=1

wm ∑i ′

F mii ′ ∑

j ′F m

jj ′ ∑k ′

F mkk ′ρi ′ j ′k ′


What is all of this good for?

In computational physics

Electrostatics, scattering, quantum mechanics, fluids, ...

Quantum mechanics (many electron atoms):

A very nasty multidimensional problem.

Collaborations with ORNL and Ohio U: a generic toolkit (MADNESS).

Applications in neuroscience? This is speculative!!!

Hierarchical modeling:

Multiresolution operator representations → effective multiscale theories.

Novel approaches for high-dimensional datasets.

PDE - based algorithms for image processing.

Wavelet-based representations of fMRI objects for algorithmic work.

...? I hope to talk to some of you today about this.



What is all of this good for?

In computational physics

Electrostatics, scattering, quantum mechanics, fluids, ...

Quantum mechanics (many electron atoms):

A very nasty multidimensional problem.

Collaborations with ORNL and Ohio U: a generic toolkit (MADNESS).

Applications in neuroscience? This is speculative!!!

Hierarchical modeling:

Multiresolution operator representations → effective multiscale theories.

Novel approaches for high-dimensional datasets.

PDE - based algorithms for image processing.

Wavelet-based representations of fMRI objects for algorithmic work.

...? I hope to talk to some of you today about this.



Outline








Python in scientific computing

Free, interactive, highly portable language.

Extremely readable syntax (“executable pseudo-code”).

Rich built-in types: lists, sets, dictionaries (hash tables), strings, ...

Very comprehensive standard library (batteries included):

Text processing, networking protocols, threading, GUIs, ...

Standard libraries for Matlab/IDL-like arrays.

Easy to wrap existing C, C++ and FORTRAN codes:

Great code reuse!


Python is growing in scientific computingNeuroimaging article by J. Millman and M. Brett

Data analysis for epilepsy surgeryJohn Hunter, Pediatric Neurology, U. Chicago

Correlation analysis of seizure data

Final location of epileptic foci for surgery


Why do I use Python?

Very complex algorithms: rich types and easy object model.

Exploratory interactive work: IPython.

Visualization: 2D and 3D

Numerics: numpy, scipy, in-house Fortran/C codes

Access to lots of third-party libraries (GUIs, networking, ...)

We need these things in scientific computing...but we don’t want to write them ourselves!



IPython: better interactive work

Why write this?

Scientific computing is inherently exploratory.

A good interactive environment is a necessity.

1 A better Python shell: object introspection, system access, . . .2 An embeddable interpreter: allow any program to be probed interactively.3 A flexible component: base layer for other interactive systems.4 High-level distributed/parallel computing.5 A component we can plug into GUIs, browser shells, etc.



IPython: better interactive work

Why write this?

Scientific computing is inherently exploratory.

A good interactive environment is a necessity.

1 A better Python shell: object introspection, system access, . . .2 An embeddable interpreter: allow any program to be probed interactively.3 A flexible component: base layer for other interactive systems.4 High-level distributed/parallel computing.5 A component we can plug into GUIs, browser shells, etc.



Matlab-like usage



Sophisticated 3d visualizations with VTK



Who uses IPython?

IPython is available for all Linux distributions, OS X and Windows.

SAGE: open source mathematics, very ambitious project (U. Washington).

PyRAF: astronomical image analysis (Space Telescope Science Institute).

CASA: Common Astronomy Software Applications (National RadioAstronomy Observatory).

Ganga: job control for the LHCb and ATLAS experiments at CERN.

PyMAD: a neutron spectrometer at the Institut Laue Langevin (Grenoble).

Pymerase: microarray gene expression databases.

Many others...



Why the popularity?

Write a useful tool...

That is highly customizable.

Keep a lively community and grow it:

Mailing lists, wiki-based website.Developers in the US and Europe.User-contributed articles and videos...

Listen to the needs of users (esp. major projects):

Add functionality they need.Adapt and evolve the design as needed.

The lesson?Solve your problem well, with enough generality and flexibility...

and you’ll solve a lot more than just your problem.



Why the popularity?











Why the popularity?











Why the popularity?











Why the popularity?











Outline








Parallel computing: why should we care?

Because reality looks like this:



Parallel computing: why should we care?

Because reality looks like this:


We can’t escape thermodynamics

The vendor’s solutions:

Multicore chips: even in your laptop.

Graphics cards for general computing: > 128 ‘processors’ per card.

High-density clusters: SiCortex (> 5000 processors in a cabinet).

We can’t escape thermodynamics

The vendor’s solutions:

Multicore chips: even in your laptop.

Graphics cards for general computing: > 128 ‘processors’ per card.

High-density clusters: SiCortex (> 5000 processors in a cabinet).


Parallel programming?

There are plenty of bad news

It is in general, extremely difficult.

Scientific productivity plummets with enormous up-front efforts.

Development, debugging, running is hard and cumbersome.

With propietary tools, licensing costs can go through the roof.

But not all is doom and gloom

Many problems are embarrassingly parallel: uncoupled components.

This is common in neuroscience:

Analyze many scans in an fMRI run.Global parameter search in generative models.SPM has plenty of opportunity for easy massive speedups.

Even not-so-embarrassingly parallel problems can be tractable...

... with the right tools.



Parallel programming?

There are plenty of bad news

It is in general, extremely difficult.

Scientific productivity plummets with enormous up-front efforts.

Development, debugging, running is hard and cumbersome.

With propietary tools, licensing costs can go through the roof.

But not all is doom and gloom

Many problems are embarrassingly parallel: uncoupled components.

This is common in neuroscience:

Analyze many scans in an fMRI run.Global parameter search in generative models.SPM has plenty of opportunity for easy massive speedups.

Even not-so-embarrassingly parallel problems can be tractable...

... with the right tools.



Network-aware IPython



Distributed/parallel computing

Think of Python as ’the CPU’

But these souped-up kernels let you talk to it conveniently.



What does IPython offer here?

Easy reuse and distribution of existing serial (‘normal’) codes.

High-level abstractions for ‘embarrassingly parallel’ problems.

Out-of-the box task farming tools.

Task farming system is low-latency

can be integrated into more complex codes.

Implement any approach to parallelism you want:

Task farming.Traditional Message Passing (MPI).Integrate hybrid codes.

Actively developed (Colorado, Berkeley).



Outline







FluidLab: a MayaVi based CFD visualization toolWith: K. Julien, P. Schmitt and B. Barrow (Applied Math, U. Colorado).

FluidLab: an Envisage project.Envisage: plugin-based system for scientific application development.


Graphics cards for numerics

Why?

Graphics cards (‘GPUs’) are basically very fast, highly parallelcomputers...

Problem: the unequally spaced FFT (Fast Fourier Transform)

Very painful, but we got our part working.

Recent hardware makes life a lot easier.

Much less of a square peg (numerical algorithm) into a round hole (a 3dgraphics language).

Possibilities for neuroscience?Real-time analysis of EEG/ECoG data (K. Koepsell, R. Canolty).

Speed up FFT and linear-algebra based algorithms.




Why?











Why?










In closing...

Better algorithms:

From the physics to the mathematics: correct and robust.Multi-scale interactions.Hierarchical models and effective theories.High dimensional problems.

Better computational tools:

Exploratory computing.Integration with GUIs, the internet, low-level codes, ...Improved access to parallel resources.

A culture of open and reproducible computational research:

Transparent workflows.Flexible tools with interchangeable components.

Ultimately

Better algorithms + better tools = better science!



In closing...

Better algorithms:






Ultimately




In closing...

Better algorithms:






Ultimately




In closing...

Better algorithms:






Ultimately




In closing...

Better algorithms:






Ultimately




EXTRA SLIDES just in case...



Sparsity also in the m ‘direction’?

The Gaussian expansion gave us separation of directions...

at the cost of a new ‘internal’ degree of freedom, the separation index m.

Do we really need all these terms?

NO: the 2-scale differences cancel most of them!



Sparsity also in the m ‘direction’?

The Gaussian expansion gave us separation of directions...

at the cost of a new ‘internal’ degree of freedom, the separation index m.

Do we really need all these terms?NO: the 2-scale differences cancel most of them!


Physics, Mathematics and Computers - Fernando Pérezfperez.org/talks/0705_berkeley.pdfPhysics/Math MRA IPython Parallel Other projects Physics, Mathematics and Computers Better algorithms

Documents