ATPESC2015 Python Dinner Talk - MCSpress3.mcs.anl.gov/atpesc/...Python-Dinner-Talk.pdfATPESC’2015’I’Challenges’of’Python’atScale’I’ 12’August2015 Science’Tools’for’Python

Python for High Performance Compu4ng

William Scullin [email protected] Leadership Compu4ng Facility Argonne Na4onal Laboratory

mailto:[email protected]

ATPESC 2015 -‐ Challenges of Python at Scale -‐ 12 August 2015

Why this talk?

2


3

“People are doing high performance computing with Python... How do we stop them?” !

- Senior Performance Engineer


Why Python?

4

ATPESC2015 -‐ Challenges of Python at Scale -‐ 12 August 2015

• Created by Guido van Rossum in 1989 • Originally a scrip4ng language for the Amoeba OS • Highly influenced by Modula-‐3, ABC, Algol, and C • It refers to both the language and to the reference

implementa4on CPython • Two major versions of the language: § Python 2 § Python 3

What’s Python?


• If you like a programming paradigm, it’s supported • Most func4ons map to what you know already • Easy to combine with other languages • Easy to keep code readable and maintainable • Lets you do just about anything without changing languages • The price is right! § No license management § Code portability § Fully Open Source § Very low learning curb

• Comes with a highly enthusias4c and helpful community

Why Use Python?


Easy to learn#include "iostream" #include "math" int main(int argc,char** argv) { int n = atoi(argv[1]); for(int i=2; i<(int) sqrt(n); i++) { p=0; while(n % i) { p+=1; n/=i; } if (p) cout << i << "^" << p << endl; } return 0; }

7

import math, sys !n = int(sys.argv[1]) for i in range(2,math.sqrt(n)): p=0 while n % i: (p,n) = (p+1,n/i) if p: print i,'^',p !sys.exit(0)


• "Baaeries included" + rich scien4fic compu4ng ecosystem • Good balance between computa4onal performance and 4me

investment § Similar performance to expensive commercial solu4ons § Many ways to op4mize cri4cal components § Only spend 4me on speed if really needed

• Tools are mostly open source and free • Strong community and commercial support op4ons. • No license management for the modules that keep people

produc4ve

Why Use Python for Scien4fic Compu4ng?


Science Tools for Python

9

General NumPySciPy GPGPU Computing PyCUDA PyOpenCL Parallel Computing PETSc PyMPI Pypar mpi4py Wrapping C/C++/Fortran SWIG Cython ctypes !

Molecular & Atomic Modeling PyMOL Biskit GPAW Geosciences GIS Python PyClimate ClimPy CDAT Bayesian StatsPyMC Optimization OpenOpt

For a more complete list: http://www.scipy.org/Topical_Software

Plotting & Visualization matplotlib VisIt Chaco MayaVi AI & Machine Learning pyem ffnet pymorph Monte hcluster Biology (inc. neuro) Brian SloppyCell NIPY PySAT

Symbolic Math SymPy !Electromagnetics PyFemax Astronomy AstroLib PySolar !Dynamic Systems Simpy PyDSTool Finite Elements SfePy !Other Languages R MATLAB

http://www.scipy.org/Topical_Software


• Low learning curve § It’s easy to write Fortran / C / C++ in Python § PEP 8 isn’t the law, just a really good ideahap://www.python.org/dev/peps/pep-‐0008/

§ Reimplementa4on of exis4ng solu4ons is way too easy-‐ if it’s important, there’s already a solu4on out there

• Easy to combine with C/C++/Fortran § there are communi4es around most major packages § really important packages have Python bindings § not all bindings are “Pythonic”

Why Not Use Python? -‐ The Language


• There’s constant revision through the PEP process

• Language maintainers strive for philosophical consistency • Backwards compa4bility is seldom guaranteed • They’re not kidding when the goal is to have only one way to do something

• features have been known to vanish e.g.: lambda • Future features are oien available in older versions to ease transi4ons

• Tim Peter’s The Zen of Python notes: In the face of ambiguity, refuse the tempta4on to guess. There should be one-‐-‐ and preferably only one -‐ obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is beaer than never. Although never is oien beaer than *right* now. !

• Language maintainers strive for “principle of least surprise” • Web folks are figh4ng for decimal numerics by default

Why Not Use Python? -‐ The Language


• It’s inefficient • Python 2.x is a true interpreter • Pure Python is interpreted line-‐by-‐line • “If you want your code to run faster, you should probably just use PyPy.” — Guido van Rossum !

• The GIL • David Beazley covers it beaer than anyone: hap://www.dabeaz.com/python/GIL.pdf

• Distu4ls • Conceived of as a way to make it easy to build and install Python modules • Really a way of thwar4ng custom linking and cross-‐compiling !

• Lots of small file I/O as part of runs !

• Debuggers and performance tools hate mixing languages

Why Not Use Python? -‐ CPython


How About A Quick Demo?

13


• N-‐dimensional homogeneous arrays (ndarray) • Universal func4ons (ufunc)

• basic math, linear algebra, FFT, PRNGs • Simple data file I/O

• text, raw binary, na4ve binary • Tools for integra4ng with C/C++/Fortran • Heavy liiing done by op4mized C / Fortran libraries

• ATLAS or MKL, UMFPACK, FFTW, etc...

14

NumPy


# Initialize with lists: array with 2 rows, 4 cols >>> import numpy as np >>> np.array([[1,2,3,4],[8,7,6,5]]) array([[1, 2, 3, 4], [8, 7, 6, 5]]) !!# Make array of evenly spaced numbers over an interval >>> np.linspace(1,100,10) array([ 1., 12., 23., 34., 45., 56., 67., 78., 89., 100.]) !!# Create and prepopulate with zeros >>> np.zeros((2,5))

Crea4ng NumPy Arrays

15


>>> a = np.array([[1,2,3,4],[9,8,7,6],[1,6,5,4]]) >>> arow = a[0,:] # get slice referencing row zero >>> arow array([1, 2, 3, 4]) >>> cols = a[:,[0,2]] # get slice referencing columns 0 and 2 >>> cols array([[1, 3], [9, 7], [1, 5]]) # NOTE: arow & cols are NOT copies, they point to the original data >>> arow[:] = 0 >>> arow array([0, 0, 0, 0]) !>>> a array([[0, 0, 0, 0], [9, 8, 7, 6], [1, 6, 5, 4]]) !# Copy data >>> copyrow = arow.copy()

Slicing Arrays

16


>>> a = np.array(([1,2,3,4],[8,7,6,5])) >>> aarray([[1, 2, 3, 4], [8, 7, 6, 5]]) # Rule 1: Dimensions of one may be prepended to either array to match the array with the greatest number of dimensions>>> a + 1 # add 1 to each element in array array([[2, 3, 4, 5], [9, 8, 7, 6]]) # Rule 2: Arrays may be repeated along dimensions of length 1 to match the size of a larger array>>> a + np.array(([1],[10])) # add 1 to 1st row, 10 to 2nd row array([[ 2, 3, 4, 5], [18, 17, 16, 15]]) >>> a**([2],[3]) # raise 1st row to power 2, 2nd to 3 array([[ 1, 4, 9, 16], [512, 343, 216, 125]])

Broadcas4ng with ufuncs apply opera4ons to many elements with a single call

17


• Extends NumPy with common scien4fic compu4ng tools • op4miza4on • addi4onal linear algebra • integra4on • interpola4on • FFT • signal and image processing • ODE solvers

• Heavy liiing done by C/Fortran code

SciPy

18


mpi4py -‐ MPI for Python

• wraps a na4ve mpi • provides all MPI2 features • well maintained • requires NumPy • insanely portable and scalable • hap://mpi4py.scipy.org/

http://mpi4py.scipy.org


How mpi4py works...

• mpi4py jobs must be launched with mpirun/mpiexec • each rank launches its own independent python interpreter

• no GIL! • each interpreter only has access to files and libraries available

locally to it, unless distributed to the ranks • communica4on is handled by MPI layer • any func4on outside of an if block specifying a rank is assumed to

be global • any limita4ons of your local MPI are present in mpi4py


mpi4py basics -‐ datatype caveats

•mpi4py can ship any serializable objects • Python objects, with the excep4on of strings and integers are

pickled • Pickling and unpickling have significant overhead • overhead impacts both senders and receivers • use the lowercase methods, eg: recv(),send()

• MPI datatypes are sent without pickling • near the speed of C • NumPy datatypes are converted to MPI datatypes • custom MPI datatypes are s4ll possible • use the capitalized methods, eg: Recv(), Send()

• When in doubt, ask if what is being processed is a memory buffer or a collec4on of pointers!


Calcula4ng pi with mpi4pyfrom mpi4py import MPIimport randomcomm = MPI.COMM_WORLDrank = comm.Get_rank()mpisize = comm.Get_size()nsamples = int(12e6/mpisize) inside = 0random.seed(rank)for i in range(nsamples): x = random.random() y = random.random() if (x*x)+(y*y)<1: inside += 1mypi = (4.0 * inside)/nsamples pi = comm.reduce(mypi, op=MPI.SUM, root=0) if rank==0: print (1.0 / mpisize)*pi


Calcula4ng pi with mpi4py and NumPyfrom mpi4py import MPI import numpy as np !comm = MPI.COMM_WORLD rank = comm.Get_rank() mpisize = comm.Get_size() nsamples = int(12e6/mpisize) !np.random.seed(rank) !xy=np.random.random((nsamples,2)) mypi=4.0*np.sum(np.sum(xy**2,1)<1)/nsamples !pi = comm.reduce(mypi, op=MPI.SUM, root=0) !if rank==0: print (1.0 / mpisize)*pi


Anyone do this in produc4on?

24


a massively parallel Python-‐C code for electronic structure calcula4ons

!


• Ab ini0o atomis4c simula4on for predic4ng material proper4es o density func4onal theory (DFT) and 4me-‐dependent density func4onal theory (TD-‐DFT) §Nobel prize in Chemistry to Walter Kohn (1998) for DFT

• Finite difference stencils on uniform real-‐space grid • Non-‐linear sparse eigenvalue problem

o ~106 grid points, ~103 eigenvalues • Wriaen in Python and C using the NumPy library • Massively parallel using MPI • Open source (GPL)

h"p://wiki.fysik.dtu.dk/gpaw J. Enkovaara et al. J. Phys.: Condens. Maaer 22, 253202 (2010)

http://wiki.fysik.dtu.dk/gpaw


GPAW Source Code Timeline


Science done with GPAW

Nature Chemistry, PRL, JACS, PNAS, PRB, ...


GPAW Strong-‐scaling Results

Ground state DFT on Blue Gene P


GPAW Strong-‐scaling Results

TD-‐DFT on Cray XT5


• Some supercompu4ng systems (BG, Cray XT) have special light-‐weight kernels on compute nodes

• Lack of "standard" features o dynamic libraries o lots of missing system calls o did we men4on all I/O is forwarded?

• Python relies heavily on dynamic loading o sta4c build of Python (including all needed C-‐extensions) is possible

omodifica4on of CPython is needed for correct namespace resolu4on

o See wiki.fysik.dtu.dk/gpaw/install/Cray/jaguar.html for some details

• Cross-‐compila4on can be challenging -‐ disaools is evil

Special opera4ng systems


• import statement triggers lots of metadata traffic o directory accesses, opening and closing files

• parallel filesystems deal well only with large files/data • There is considerably amount of imports already during Python ini4aliza4on (and yes, we trim site.py and the module search path) o Ini4aliza4on overheads do not show up in the Python 4mers

• With > 1000 processes problem can be severe even in produc4on calcula4ons o with 8 racks (~32 000 cores) on Blue Gene /P Python start-‐up 4me can be 45 minutes!

Python's import mechanism and parallel scalability


• Possible solu4ons (all are sort of ugly) o Put all the Python modules on a ramdisk oHack CPython -‐ only single process reads (module) files and broadcasts data to others with MPI

o develop extreme pa4ence

Python's import mechanism and parallel scalability


Ques4ons?


Acknowledgments This work is supported in part by the resources of the Argonne Leadership Computing Facility at Argonne National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under contract DE-AC02-06CH11357. Extended thanks to • CSC • Northwestern University • De Paul University • Sameer Shende, ParaTools, Inc. • NumFocus for their continued support and sponsorship of

SciPy and NumPy • Lisandro Dalcin for his work on mpi4py and petsc4py • ChiPy

ATPESC2015 Python Dinner Talk - MCSpress3.mcs.anl.gov/atpesc/...Python-Dinner-Talk.pdfATPESC’2015’I’Challenges’of’Python’atScale’I’ 12’August2015 Science’Tools’for’Python

Documents