Lecture 5 Barriers and MPI Introduction Topics Topics Barriers Uses implementations MPI Introduction Readings – Semaphore handout Readings – Semaphore handout dropboxed dropboxed January 24, 2012 CSCE 713 Advanced Computer Architecture
Lecture 5Barriers and MPI Introduction
Lecture 5Barriers and MPI Introduction
Topics Topics Barriers
Uses implementations
MPI Introduction
Readings – Semaphore handout Readings – Semaphore handout dropboxeddropboxed
January 24, 2012
CSCE 713 Advanced Computer Architecture
– 2 –CSCE 713 Spring 2012
OverviewOverviewLast TimeLast Time
Posix Pthreads: create, join, exit, mutexes /class/csce713-006 Code and Data
Readings for todayReadings for today http://csapp.cs.cmu.edu/public/1e/public/ch9-preview.pdf
NewNew Website alive and kicking; dropbox too! From Last Lecture’s slides: Gauss-Seidel Method, Barriers,
Threads Assignment Next time performance evaluation, barriers and MPI intro
– 3 –CSCE 713 Spring 2012
Threads programming AssignmentThreads programming Assignment
1.1. Matrix addition (embarassingly parallel)Matrix addition (embarassingly parallel)
2.2. VersionsVersionsa. Sequential
b. Sequential with blocking factor
c. Sequential Read without conversions
d. Multi threaded passing number of threads as command line argument (args.c code should be distributed as an example)
3.3. Plot of several runsPlot of several runs
4.4. Next timeNext time
5.5. Test Data in /class/csce713-006/DataTest Data in /class/csce713-006/Data• A100, A400, B100, B400
– 4 –CSCE 713 Spring 2012
Next few slides are mostly from Parallel Programming Next few slides are mostly from Parallel Programming by Pachecoby Pacheco
/class/csce713-006/Code/Pacheco contains the code /class/csce713-006/Code/Pacheco contains the code (again only for use with this course do not distribute)(again only for use with this course do not distribute)
– 5 –CSCE 713 Spring 2012
Recall use of mutex to control access to a variableRecall use of mutex to control access to a variable
Copyright © 2010, Elsevier Inc. All rights Reserved
– 6 –CSCE 713 Spring 2012
Barriers - synchronizing threadsBarriers - synchronizing threads
Synchronizing threads after a period of computationSynchronizing threads after a period of computation
• No thread proceeds until all others have reached the No thread proceeds until all others have reached the barrierbarrier
E.g., last time iteration of Gauss-Siedel, sync check for E.g., last time iteration of Gauss-Siedel, sync check for convergenceconvergence
Examples of barrier uses and implementationsExamples of barrier uses and implementations
1.1. Using barriers for testing convergence, i.e. satisfying Using barriers for testing convergence, i.e. satisfying a completion criteriaa completion criteria
2.2. Using barriers to time the slowest threadUsing barriers to time the slowest thread
3.3. Using barriers for debuggingUsing barriers for debugging
– 7 –CSCE 713 Spring 2012
Using barriers for testing convergence, i.e. satisfying a completion criteriaUsing barriers for testing convergence, i.e. satisfying a completion criteriaSlide 43 of Lecture 3 (slightly modified)Slide 43 of Lecture 3 (slightly modified)
General computationGeneral computation
Stop when all xStop when all xii from this iteration calculated from this iteration calculated
ii
i
ii
i,
n
j1j
jj,
a
xac
x
n
1i
2i )x- (berror i
– 8 –CSCE 713 Spring 2012
Using barriers to time the slowest threadUsing barriers to time the slowest thread
Copyright © 2010, Elsevier Inc. All rights Reserved
– 9 –CSCE 713 Spring 2012
Using barriers for debuggingUsing barriers for debugging
Copyright © 2010, Elsevier Inc. All rights Reserved
– 10 –CSCE 713 Spring 2012
Barrier ImplementationsBarrier Implementations
1.1. Mutex , threadCounter, busy-waitsMutex , threadCounter, busy-waits
2.2. Mutex plus barrier semaphoreMutex plus barrier semaphore
3.3. Mutex and condition variablesMutex and condition variables
– 11 –CSCE 713 Spring 2012
Busy-waiting and a MutexBusy-waiting and a Mutex
Implementing a barrier using busy-waiting and a mutex Implementing a barrier using busy-waiting and a mutex is straightforward.is straightforward.
We use a shared counter protected by the mutex.We use a shared counter protected by the mutex.
When the counter indicates that every thread has When the counter indicates that every thread has entered the critical section, threads can leave the entered the critical section, threads can leave the critical section.critical section.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 12 –CSCE 713 Spring 2012
Busy-waiting and a MutexBusy-waiting and a Mutex
Copyright © 2010, Elsevier Inc. All rights Reserved
We need one counter variable for each
instance of the barrier,
otherwise problemsare likely to occur.
– 13 –CSCE 713 Spring 2012
Implementing a barrier with semaphoresImplementing a barrier with semaphores
Copyright © 2010, Elsevier Inc. All rights Reserved
– 14 –CSCE 713 Spring 2012
Condition VariablesCondition Variables
A condition variable is a data object that allows a A condition variable is a data object that allows a thread to suspend execution until a certain event or thread to suspend execution until a certain event or condition occurs.condition occurs.
When the event or condition occurs another thread can When the event or condition occurs another thread can signal the thread to “wake up.”signal the thread to “wake up.”
A condition variable is always associated with a mutex.A condition variable is always associated with a mutex.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 15 –CSCE 713 Spring 2012
Condition VariablesCondition Variables
Copyright © 2010, Elsevier Inc. All rights Reserved
– 16 –CSCE 713 Spring 2012
Implementing a barrier with condition variablesImplementing a barrier with condition variables
Copyright © 2010, Elsevier Inc. All rights Reserved
– 17 –CSCE 713 Spring 2012
POSIX Semaphores POSIX Semaphores
$ man -k semaphore$ man -k semaphoresem_close (3) - close a named semaphoresem_close (3) - close a named semaphoresem_destroy (3) - destroy an unnamed semaphoresem_destroy (3) - destroy an unnamed semaphoresem_getvalue (3) - get the value of a semaphoresem_getvalue (3) - get the value of a semaphoresem_init (3) - initialize an unnamed semaphoresem_init (3) - initialize an unnamed semaphoresem_open (3) - initialize and open a named semaphoresem_open (3) - initialize and open a named semaphoresem_overview (7) - Overview of POSIX semaphoressem_overview (7) - Overview of POSIX semaphoressem_post (3) - unlock a semaphoresem_post (3) - unlock a semaphoresem_timedwait (3) - lock a semaphoresem_timedwait (3) - lock a semaphoresem_trywait (3) - lock a semaphoresem_trywait (3) - lock a semaphoresem_unlink (3) - remove a named semaphoresem_unlink (3) - remove a named semaphoresem_wait (3) - lock a semaphoresem_wait (3) - lock a semaphore
$ man –s 7 semaphores$ man –s 7 semaphoresNo manual entry for semaphore in section 7No manual entry for semaphore in section 7
– 18 –CSCE 713 Spring 2012
Implementing a barrier with condition variablesImplementing a barrier with condition variables
Copyright © 2010, Elsevier Inc. All rights Reserved
– 19 –CSCE 713 Spring 2012
Pthread BarriersPthread Barriers
// Barrier variable pthread_barrier_t barr;// Barrier variable pthread_barrier_t barr;
// Barrier initialization // Barrier initialization
if(pthread_barrier_init(&barr, NULL, THREADS)) if(pthread_barrier_init(&barr, NULL, THREADS)) { printf("Could not create a barrier\n"); { printf("Could not create a barrier\n");
return -1; return -1;
} }
// Create Threads// Create Threads
for(int i = 0; i < THREADS; ++i) { for(int i = 0; i < THREADS; ++i) {
if(pthread_create(&thr[i], NULL, &entry_point, (void*)i)) if(pthread_create(&thr[i], NULL, &entry_point, (void*)i)) ……
– 20 –CSCE 713 Spring 2012
Wait on Barrier in thread functionWait on Barrier in thread function
// Synchronization point // Synchronization point
int rc = pthread_barrier_wait(&barr); int rc = pthread_barrier_wait(&barr);
if(rc != 0 && rc !=PTHREAD_BARRIER_SERIAL_THREAD){ if(rc != 0 && rc !=PTHREAD_BARRIER_SERIAL_THREAD){
printf("Could not wait on barrier\n"); printf("Could not wait on barrier\n");
exit(-1); exit(-1);
}}
– 21 –CSCE 713 Spring 2012
System V Semaphores (Sets)System V Semaphores (Sets)
In System V IPC there are semaphore setsIn System V IPC there are semaphore setsCommandsCommands• ipcs - provide information on ipc facilities ipcs - provide information on ipc facilities • ipcrm - remove a message queue, semaphore set or shared ipcrm - remove a message queue, semaphore set or shared
memory segment etc.memory segment etc.
System Calls:System Calls:semctl (2) - semaphore control operationssemctl (2) - semaphore control operationssemget (2) - get a semaphore set identifiersemget (2) - get a semaphore set identifiersemop (2) - semaphore operationssemop (2) - semaphore operationssemtimedop (2) - semaphore operationssemtimedop (2) - semaphore operations
– 22 –CSCE 713 Spring 2012
MPI IntroductionMPI Introduction
– 23 –CSCE 713 Spring 2012
A distributed memory systemA distributed memory system
Copyright © 2010, Elsevier Inc. All rights Reserved
– 24 –CSCE 713 Spring 2012
A shared memory systemA shared memory system
Copyright © 2010, Elsevier Inc. All rights Reserved
– 25 –CSCE 713 Spring 2012
A tree-structured global sumA tree-structured global sum
Copyright © 2010, Elsevier Inc. All rights Reserved
– 26 –CSCE 713 Spring 2012
An alternative tree-structured global sumAn alternative tree-structured global sum
Copyright © 2010, Elsevier Inc. All rights Reserved
– 27 –CSCE 713 Spring 2012
Identifying MPI processesIdentifying MPI processes
Common practice to identify processes by nonnegative Common practice to identify processes by nonnegative integer ranks.integer ranks.
pp processes are numbered processes are numbered 0, 1, 2, .. p-10, 1, 2, .. p-1
Copyright © 2010, Elsevier Inc. All rights Reserved
– 28 –CSCE 713 Spring 2012
Our first MPI programOur first MPI program
Copyright © 2010, Elsevier Inc. All rights Reserved
– 29 –CSCE 713 Spring 2012
CompilationCompilation
mpicc -g -Wall -o mpi_hello mpi_hello.c
– 30 –CSCE 713 Spring 2012
ExecutionExecution
Copyright © 2010, Elsevier Inc. All rights Reserved
mpiexec -n <number of processes> <executable>
mpiexec -n 1 ./mpi_hello
mpiexec -n 4 ./mpi_hello
run with 1 process
run with 4 processes
– 31 –CSCE 713 Spring 2012
ExecutionExecution
Copyright © 2010, Elsevier Inc. All rights Reserved
mpiexec -n 1 ./mpi_hello
mpiexec -n 4 ./mpi_hello
Greetings from process 0 of 1 !
Greetings from process 0 of 4 !
Greetings from process 1 of 4 !
Greetings from process 2 of 4 !
Greetings from process 3 of 4 !
– 32 –CSCE 713 Spring 2012
MPI ProgramsMPI ProgramsWritten in C.Written in C.
Has main. Uses stdio.h, string.h, etc.
Need to add Need to add mpi.hmpi.h header file. header file.
Identifiers defined by MPI start with “MPI_”.Identifiers defined by MPI start with “MPI_”.
First letter following underscore is uppercase.First letter following underscore is uppercase. For function names and MPI-defined types. Helps to avoid confusion.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 33 –CSCE 713 Spring 2012
MPI ComponentsMPI Components
MPI_InitMPI_Init Tells MPI to do all the necessary setup.
MPI_FinalizeMPI_Finalize Tells MPI we’re done, so clean up anything allocated for this
program.
Copyright © 2010, Elsevier Inc. All rights Reserved
– 34 –CSCE 713 Spring 2012
Basic OutlineBasic Outline
Copyright © 2010, Elsevier Inc. All rights Reserved
– 35 –CSCE 713 Spring 2012
Communicators Communicators A collection of processes that can send messages to A collection of processes that can send messages to
each other.each other.
MPI_Init defines a communicator that consists of all the MPI_Init defines a communicator that consists of all the processes created when the program is started.processes created when the program is started.
Called Called MPI_COMM_WORLDMPI_COMM_WORLD..
Copyright © 2010, Elsevier Inc. All rights Reserved
– 36 –CSCE 713 Spring 2012
CommunicatorsCommunicators
Copyright © 2010, Elsevier Inc. All rights Reserved
number of processes in the communicator
my rank (the process making this call)