(Open) MPI, Parallel Computing, Life, the Universe, and Everything

{Open} MPI, Parallel Computing, Life, the Universe, and Everything

Dr. Jeffrey M. Squyres

November 7, 2013

Open MPI

PACX-MPI LAM/MPI

LA-MPI FT-MPI

Sun CT 6

Project founded in 2003 after intense

discussions between multiple open source MPI implementations

Open_MPI_Init()

shell$ svn log https://svn.open-mpi.org/svn/ompi -r 1 ------------------------------------------------------------------------ r1 | jsquyres | 2003-11-22 11:36:58 -0500 (Sat, 22 Nov 2003) | 2 lines First commit ------------------------------------------------------------------------ shell$

Open_MPI_Current_status()

shell$ svn log https://svn.open-mpi.org/svn/ompi -r HEAD ------------------------------------------------------------------------ r29619 | brbarret | 2013-11-06 09:14:24 -0800 (Wed, 06 Nov 2013) | 2 lines update ignore file ------------------------------------------------------------------------ shell$

Open MPI 2014 membership

13 members, 15 contributors, 2 partners

Fun stats

•  ohloh.net says: §  819,741 lines of code §  Average 10-20

committers at a time §  “Well-commented

source code”

•  I rank in top-25 ohloh stats for: §  C §  Automake §  Shell script §  Fortran (…ouch)

Current status

•  Version 1.6.5 / stable series § Unlikely to see another release

•  Version 1.7.3 / feature series §  v1.7.4 due (hopefully) by end of 2013 §  Plan to transition to v1.8 in Q1 2014

MPI conformance

•  MPI-2.2 conformant as of v1.7.3 §  Finally finished several 2.2 issues that no one

really cares about •  MPI-3 conformance just missing new RMA

§  Tracked on wiki: https://svn.open-mpi.org/trac/ompi/wiki/MPIConformance

§ Hope to be done by v1.7.4

New MPI-3 features

•  Mo’ betta Fortran bindings §  You should “use mpi_f08”. Really.

•  Matched probe •  Sparse and neighborhood collectives •  “MPI_T” tools interface •  Nonblocking communicator duplication •  Noncollective communicator creation •  Hindexed block datatype

New Open MPI features

•  Better support for more runtime systems §  PMI2 scalability, etc.

•  New generalized processor affinity system •  Better CUDA support •  Java MPI bindings (!) •  Transports:

§ Cisco usNIC support § Mellanox MXM2 and hcoll support §  Portals 4 support

My new favorite random feature

•  mpirun CLI option <tab> completion §  Bash and zsh § Contributed by Nathan Hjelm, LANL

shell$ mpirun --mca btl_usnic_<tab> btl_usnic_cq_num -- Number of completion queue!btl_usnic_eager_limit -- Eager send limit (0 = use !btl_usnic_if_exclude -- Comma-delimited list of de!btl_usnic_if_include -- Comma-delimited list of de!btl_usnic_max_btls -- Maximum number of usNICs t!btl_usnic_mpool -- Name of the memory pool to!btl_usnic_prio_rd_num -- Number of pre-posted prior!btl_usnic_prio_sd_num -- Maximum priority send desc!btl_usnic_priority_limit -- Max size of "priority" mes!btl_usnic_rd_num -- Number of pre-posted recei!btl_usnic_retrans_timeout -- Number of microseconds bef!btl_usnic_rndv_eager_limit -- Eager rendezvous limit (0 !btl_usnic_sd_num -- Maximum send descriptors t!

Two features to discuss in detail…

1. “MPI_T” interface 2. Flexible process affinity system

MPI_T interface

MPI_T interface

•  Added in MPI-3.0 •  So-called “MPI_T” because all the

functions start with that prefix §  T = tools

•  APIs to get/set MPI implementation values § Control variables (e.g., implementation

tunables) §  Performance variables (e.g., run-time stats)

MPI_T control variables (“cvar”)

•  Another interface to MCA param values •  In addition to existing methods:

§ mpirun CLI options §  Environment variables § Config file(s)

•  Allows tools / applications to programmatically list all OMPI MCA params

MPI_T cvar example

•  MPI_T_cvar_get_num() § Returns the number of control variables

•  MPI_T_cvar_get_info(index, …) returns: §  String name and description §  Verbosity level (see next slide) §  Type of the variable (integer, double, etc.) §  Type of MPI object (communicator, etc.) §  “Writability” scope

Verbosity levels

Level name Level description

USER_BASIC Basic information of interest to users USER_DETAIL Detailed information of interest to users USER_ALL All remaining information of interest to users TUNER_BASIC Basic information of interest for tuning TUNER_DETAIL Detailed information of interest for tuning TUNER_ALL All remaining information of interest to tuning MPIDEV_BASIC Basic information for MPI implementers MPIDEV_DETAIL Detailed information for MPI implementers MPIDEV_ALL All remaining information for MPI implementers

Open MPI interpretation of verbosity levels

1.  User §  Parameters required

for correctness §  As few as possible

2.  Tuner §  Tweak MPI

performance §  Resource levels, etc.

3.  MPI developer §  For Open MPI devs

1.  Basic Even for less-advanced users and tuners

2.  Detailed Useful but you won’t need to change them often

3.  All Anything else

“Writeability” scope

Level name Level description

CONSTANT Read-only, constant value READONLY Read-only, but the value may change LOCAL Writing is local operation GROUP Writing must be done as a group, and all values

must be consistent GROUP_EQ Writing must be done as a group, and all values

must be exactly the same ALL Writing must be done by all processes, and all

values must be consistent ALL_EQ Writing must be done by all processes, and all

values must be exactly the same

Reading / writing a cvar

•  MPI_T_cvar_handle_alloc(index, handle, …) §  Allocates an MPI_T handle §  Binds it to a specific MPI handle (e.g., a

communicator), or BIND_NO_OBJECT •  MPI_T_cvar_read(handle, buf) •  MPI_T_cvar_write(handle, buf)

à OMPI has very, very few writable control variables after MPI_INIT

MPI_T Performance variables (“pvar”)

•  New information available from OMPI § Run-time statistics of implementation details §  Similar interface to control variables

•  Not many available in OMPI yet •  Cisco usnic BTL exports 24 pvars

§  Per usNIC interface §  Stats about underlying network

(more details to be provided in usNIC talk)

Process affinity system

Locality matters

•  Goals: § Minimize data transfer distance § Reduce network congestion and contention

• …this also matters inside the server, too!

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

Intel Xeon E5-2690 (“Sandy Bridge”) 2 sockets, 8 cores, 64GB per socket

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

Intel Xeon E5-2690 (“Sandy Bridge”) 2 sockets, 8 cores, 64GB per socket

1G NICs

10G NICs

10G NICs

L1 and L2

Shared L3

Hyperthreading enabled

The intent of this work is to provide a mechanism that allows users to explore the process-placement space

within the scope of their own applications.

A user’s playground

Two complimentary systems

•  Simple § mpirun --bind-to [ core | socket | … ] … § mpirun --by[ node | slot | … ] … § …etc.

•  Flexible §  LAMA: Locality Aware Mapping Algorithm

LAMA

•  Supports a wide range of regular mapping patterns § Drawn from much prior work § Most notably, heavily inspired by BlueGene/P

and /Q mapping systems

Launching MPI applications

•  Three steps in MPI process placement 1.  Mapping 2.  Ordering 3.  Binding

•  Let's discuss how these work in Open MPI

1. Mapping

•  Create a layout of processes-to-resources

Server Server Server Server




MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI








Mapping

•  MPI's runtime must create a map, pairing processes-to-processors (and memory).

•  Basic technique: § Gather hwloc topologies from allocated nodes. § Mapping agent then makes a plan for which

resources are assigned to processes

Mapping agent

•  Act of planning mappings: §  Specify which process will be launched on

each server §  Identify if any hardware resource will be

oversubscribed •  Processes are mapped to the resolution of

a single processing unit (PU) §  Smallest unit of allocation: hardware thread §  In HPC, usually the same as a processor core

Oversubscription

•  Common / usual definition: § When a single PU is assigned more than one

process •  Complicating the definition:

§  Some application may need more than one PU per process (multithreaded applications)

•  How can the user express what their application means by “oversubscription”?

2. Ordering: by “slot”

Assigning MCW ranks to mapped processes

0 1 2 3

4 5 6 7

8 9 10 11

12 13 14 15

16 17 18 19

20 21 22 23

24 25 26 27

28 29 30 31

32 33 18 19

36 37 22 23

40 41 26 27

44 45 30 31

48 49 50 51

4 5 6 7

8 9 10 11

12 13 14 15

64 65 66 67

20 21 22 23

24 25 26 27

28 29 30 31

80 81 18 19

36 37 22 23

40 41 26 27

44 45 30 31

2. Ordering: by node

Assigning MCW ranks to mapped processes

0 16 32 48

64 80 96 112

128 144 160 176

192 208 224 240

1 17 33 49

65 81 97 113

129 145 161 177

193 209 225 241

2 18 18 19

66 82 22 23

130 146 26 27

194 210 30 31

4 20 36 52

4 5 6 7

8 9 10 11

12 13 14 15

5 23 37 53

20 21 22 23

24 25 26 27

28 29 30 31

6 81 18 19

36 37 22 23

40 41 26 27

44 45 30 31

Ordering

•  Each process must be assigned a unique rank in MPI_COMM_WORLD

•  Two common types of ordering: §  natural

•  The order in which processes are mapped determines their rank in MCW

§  sequential •  The processes are sequentially numbered starting

at the first processing unit, and continuing until the last processing unit

3. Binding

•  Launch processes and enforce the layout Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

0 1 2 3 4 5 6 7

8 9 10 11 12 13 14 15

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

16 17 18 19 20 21 22 23

24 25 26 27 28 29 30 31

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

32 33 34 3 4 5 6 7

40 41 42 11 12 13 14 15

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

0 1 2 3 4 5 6 7

8 9 10 11 12 13 14 15

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

0 1 2 3 4 5 6 7

8 9 10 11 12 13 14 15

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

0 1 2 3 4 5 6 7

8 9 10 11 12 13 14 15

Binding

•  Process-launching agent working with the OS to limit where each process can run: 1.  No restrictions 2.  Limited set of restrictions 3.  Specific resource restrictions

•  “Binding width” §  The number of PUs to which a process is

bound

Command Line Interface (CLI)

•  4 levels of abstraction for the user §  Level 1: None §  Level 2: Simple, common patterns §  Level 3: LAMA process layout regular patterns §  Level 4: Irregular patterns (not described in

this talk)

CLI: Level 1 (none)

•  No mapping or binding options specified § May or may not specify the number of

processes to launch (-np) §  If not specified, default to the number of cores

available in the allocation § One process is mapped to each core in the

system in a "by-core" style §  Processes are not bound

• …for backwards compatibility reasons L

CLI: Level 2 (common)

•  Simple, common patterns for mapping and binding §  Specify mapping pattern with

•  --map-by X (e.g., --map-by socket) §  Specify binding option with:

•  --bind-to Y (e.g., --bind-to core) §  All of these options are translated to Level 3

options for processing by LAMA (full list of X / Y values shown later)

CLI: Level 3 (regular patterns)

•  LAMA process layout regular patterns §  Power users wanting something unique for

their application §  Four MCA run-time parameters

•  rmaps_lama_map: Mapping process layout •  rmaps_lama_bind: Binding width •  rmaps_lama_order: Ordering of MCW ranks •  rmaps_lama_mppr: Maximum allowable number of

processes per resource (oversubscription)

rmaps_lama_map (map)

•  Takes as an argument the "process layout" §  A series of nine tokens

•  allowing 9! (362,880) mapping permutation options.

§  Preferred iteration order for LAMA •  innermost iteration specified first •  outermost iteration specified last

Example system

2 servers (nodes), 4 sockets, 2 cores, 2 PUs Node 0

Node 1

Core 1H0H1

Socket 0Core 0H0H1

016

420

Core 1H0H1

Socket 1Core 0H0H1

117

521

Core 1H0H1

Socket 2Core 0H0H1

218

622

Core 1H0H1

Socket 3Core 0H0H1

319

723

Core 1H0H1

Socket 0Core 0H0H1

8 12Core 1H0H1

Socket 1Core 0H0H1

9 13

Core 1H0H1

Socket 2Core 0H0H1

10 14Core 1H0H1

Socket 3Core 0H0H1

11 15


•  map=scbnh (a.k.a., by socket, then by core) Node 0

Node 1

Core 1H0H1

Socket 0Core 0H0H1

016

420

Core 1H0H1

Socket 1Core 0H0H1

117

521

Core 1H0H1

Socket 2Core 0H0H1

218

622

Core 1H0H1

Socket 3Core 0H0H1

319

723

Core 1H0H1

Socket 0Core 0H0H1

8 12Core 1H0H1

Socket 1Core 0H0H1

9 13

Core 1H0H1

Socket 2Core 0H0H1

10 14Core 1H0H1

Socket 3Core 0H0H1

11 15

Step 1: Traverse sockets



Node 1

Core 1H0H1

Socket 0Core 0H0H1

016

420

Core 1H0H1

Socket 1Core 0H0H1

117

521

Core 1H0H1

Socket 2Core 0H0H1

218

622

Core 1H0H1

Socket 3Core 0H0H1

319

723

Core 1H0H1

Socket 0Core 0H0H1

8 12Core 1H0H1

Socket 1Core 0H0H1

9 13

Core 1H0H1

Socket 2Core 0H0H1

10 14Core 1H0H1

Socket 3Core 0H0H1

11 15

Step 2: Ran out of sockets, so now traverse cores



Node 1

Core 1H0H1

Socket 0Core 0H0H1

016

420

Core 1H0H1

Socket 1Core 0H0H1

117

521

Core 1H0H1

Socket 2Core 0H0H1

218

622

Core 1H0H1

Socket 3Core 0H0H1

319

723

Core 1H0H1

Socket 0Core 0H0H1

8 12Core 1H0H1

Socket 1Core 0H0H1

9 13

Core 1H0H1

Socket 2Core 0H0H1

10 14Core 1H0H1

Socket 3Core 0H0H1

11 15

Step 3: Now traverse boards (but there aren’t any)



Node 1

Core 1H0H1

Socket 0Core 0H0H1

016

420

Core 1H0H1

Socket 1Core 0H0H1

117

521

Core 1H0H1

Socket 2Core 0H0H1

218

622

Core 1H0H1

Socket 3Core 0H0H1

319

723

Core 1H0H1

Socket 0Core 0H0H1

8 12Core 1H0H1

Socket 1Core 0H0H1

9 13

Core 1H0H1

Socket 2Core 0H0H1

10 14Core 1H0H1

Socket 3Core 0H0H1

11 15

Step 4: Now traverse server nodes



Node 1

Core 1H0H1

Socket 0Core 0H0H1

016

420

Core 1H0H1

Socket 1Core 0H0H1

117

521

Core 1H0H1

Socket 2Core 0H0H1

218

622

Core 1H0H1

Socket 3Core 0H0H1

319

723

Core 1H0H1

Socket 0Core 0H0H1

8 12Core 1H0H1

Socket 1Core 0H0H1

9 13

Core 1H0H1

Socket 2Core 0H0H1

10 14Core 1H0H1

Socket 3Core 0H0H1

11 15

Step 5: After repeating s, c, and b on server node 2, traverse hardware threads

rmaps_lama_bind (bind)

•  “Binding width" and layer •  Example: bind=3c (3 cores) Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

bind = 3c

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013


•  “Binding width" and layer •  Example: bind=2s (2 sockets) Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

bind = 2s

bind = 2s


•  “Binding width" and layer •  Example: bind=12 (all PUs in an L2)

bind = 12


•  “Binding width" and layer •  Example: bind=1N (all PUs in NUMA locality)

bind = 1N

rmaps_lama_order (order)

•  Select which ranks are assigned to processes in MCW

•  There are other possible orderings, but no one has asked for them yet…

Natural order for map-by-node (default)

Sequential order for any mapping

rmaps_lama_mppr (mppr)

•  mppr (mip-per) sets the Maximum number of allowable Processes Per Resource § User-specified definition of oversubscription

•  Comma-delimited list of <#:resource>!§  1:c à At most one process per core §  1:c,2:s à At most one process per core, and

at most two processes per socket

MPPR

§  1:c à At most one process per core Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

MPPR

§  1:c,2:s à At most one process per core and two processes per socket

Machine (128GB)

NUMANode P#0 (64GB)

Socket P#0

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#0

PU P#16

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#1

PU P#17

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#2

PU P#18

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#3

PU P#19

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#4

PU P#20

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#5

PU P#21

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#6

PU P#22

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#7

PU P#23

PCI 8086:1521

eth0

PCI 8086:1521

eth1

PCI 8086:1521

eth2

PCI 8086:1521

eth3

PCI 1137:0043

eth4

PCI 1137:0043

eth5

PCI 102b:0522

NUMANode P#1 (64GB)

Socket P#1

L3 (20MB)

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#0

PU P#8

PU P#24

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#1

PU P#9

PU P#25

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#2

PU P#10

PU P#26

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#3

PU P#11

PU P#27

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#4

PU P#12

PU P#28

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#5

PU P#13

PU P#29

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#6

PU P#14

PU P#30

L2 (256KB)

L1d (32KB)

L1i (32KB)

Core P#7

PU P#15

PU P#31

PCI 1000:005b

sda sdb

PCI 1137:0043

eth6

PCI 1137:0043

eth7

Indexes: physical

Date: Mon Jan 28 10:51:26 2013

Level 2 to Level 3 chart

Remember the prior example?

•  -np 24 -mppr 2:c -map scbnh Node 0

Node 1

Core 1H0H1

Socket 0Core 0H0H1

016

420

Core 1H0H1

Socket 1Core 0H0H1

117

521

Core 1H0H1

Socket 2Core 0H0H1

218

622

Core 1H0H1

Socket 3Core 0H0H1

319

723

Core 1H0H1

Socket 0Core 0H0H1

8 12Core 1H0H1

Socket 1Core 0H0H1

9 13

Core 1H0H1

Socket 2Core 0H0H1

10 14Core 1H0H1

Socket 3Core 0H0H1

11 15

Same example, different mapping

•  -np 24 -mppr 2:c -map nbsch Node 0

Node 1

Core 1H0H1

Socket 0Core 0H0H1

016

8Core 1H0H1

Socket 1Core 0H0H1

218

10

Core 1H0H1

Socket 2Core 0H0H1

420

12Core 1H0H1

Socket 3Core 0H0H1

622

14

Core 1H0H1

Socket 0Core 0H0H1

1 9Core 1H0H1

Socket 1Core 0H0H1

3 11

Core 1H0H1

Socket 2Core 0H0H1

5 13Core 1H0H1

Socket 3Core 0H0H1

7 15

17 19

21 23

•  Displays prettyprint representation of the binding actually used for each process. §  Visual feedback = quite helpful when exploring

mpirun -np 4 --mca rmaps lama --mca rmaps_lama_bind 1c --mca rmaps_lama_map nbsch --mca rmaps_lama_mppr 1:c --report-bindings hello_world!

MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..]!MCW rank 1 bound to socket 1[core 8[hwt 0-1]]: [../../../../../../../..][BB/../../../../../../..]!MCW rank 2 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../..][../../../../../../../..]!MCW rank 3 bound to socket 1[core 9[hwt 0-1]]: [../../../../../../../..][../BB/../../../../../..]!

Report bindings

Feedback

•  Available in Open MPI v1.7.2 (and later)

•  Open questions to users: §  Are more flexible ordering options useful? § What common mapping patterns are useful? § What additional features would you like to

see?

Thank you

(Open) MPI, Parallel Computing, Life, the Universe, and Everything

Technology

mpi conformance mpi

cvar mpi

new mpi

mpi implementersmpidev

mpi developer

socalled mpi

new open mpi

open mpi devs1