csinparallel.org
Patterns and Exemplars: Compelling Strategies for Teaching Parallel and
Distributed Computing to CS Undergraduates
Libby Shoop Joel Adams Dick Brown
csinparallel.org
Today’s messages
• Parallel Design Patterns provide an established, practical set of principles for teaching PDC
• “Exemplar” example applications with multiple implemented solutions provide motivation for students and teaching materials for instructors
• Patterns and Exemplars fit together naturally and are ready for deployment
csinparallel.org
Parallel Design Patterns
• Following on the original Gang of Four design patterns work
Active work on parallel design patterns and parallel pattern languages:• Catalog parallel patterns used in solutions and
describe a methodology for using the pattern
csinparallel.org
Past Work• Lea :
– Java Concurrency Patterns book• Mattson, Saunders, and Massingil :
– PPLP book• Ralph Johnson et al. :
– Parallel Programming Patterns online; books of Visual C++, .NET examples
• Oretega-Arjona book• McCool, Reinders, and Robison book
• Kreutzer, Mattson, et al. : – Our Pattern Language (OPL) online
• ParaPLoP Workshop on Parallel Programming Patterns
ParaPLoP ‘10
1999
2004
2010 2011
2010 2012
csinparallel.org
Pattern Approach
• Using existing design knowledge when designing new parallel programs
• Leads to parallel software systems that are:– modular, adaptable, understandable and evolve
easily
• Also provides an effective problem-solving framework and a guide for teaching about good parallel solutions
csinparallel.org
PATTERNLETS
csinparallel.org
Patternlets…
… are minimalist, scalable, executable programs, each illustrating a particular pattern’s behavior:– Minimalist so that students can grasp the concept
without non-essential details getting in the way– Scalable so that students see different behaviors as
the number of threads changes– Executable so that
• Instructors can use it in a live-coding demo• Students can use it in a hands-on exercise
Patternlets let students see the pattern in action
csinparallel.org
Existing Patternlets (so far)
• OpenMP– Fork-Join– SPMD– Master-Worker– Parallel For Loop (blocks)– Parallel For Loop (stripes)– Reduction– Private– Atomic– Critical– Critical2– Sections– Barrier
• MPI– SPMD– Master-Worker– Message Passing– Parallel For Loop (stripes)– Parallel For Loop (blocks)– Broadcast– Reduction– Scatter– Gather– Barrier
MPI PatternletsOpenMP Patternlets
csinparallel.org
/* masterWorker.c (MPI) … */
#include <stdio.h>#include <mpi.h>
int main(int argc, char** argv) { int id = -1, numProcs= -1, length = -1; char hostName[MPI_MAX_PROCESSOR_NAME];
MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &id); MPI_Comm_size(MPI_COMM_WORLD, &numProcs); MPI_Get_processor_name (hostName, &length);
if ( id == 0 ) { // process with ID == 0 is the master printf("Greetings from the master, #%d (%s) of %d processes\n”, id, hostName, numProcs); } else { // processes with IDs > 0 are workers printf("Greetings from a worker, #%d (%s) of %d processes\n”, id, hostName, numProcs); }
MPI_Finalize(); return 0;}
csinparallel.org
Sample Executions$ mpirun -np 1 ./masterWorkerGreetings from the master, #0 (node-01) of 1 processes
$ mpirun –np 8 ./masterWorkerGreetings from the master, #0 (node-01) of 8 processesGreetings from a worker, #1 (node-02) of 8 processesGreetings from a worker, #5 (node-06) of 8 processesGreetings from a worker, #3 (node-04) of 8 processesGreetings from a worker, #4 (node-05) of 8 processesGreetings from a worker, #7 (node-08) of 8 processesGreetings from a worker, #2 (node-03) of 8 processesGreetings from a worker, #6 (node-07) of 8 processes
csinparallel.org
/* masterWorker.c (OpenMP) … */
#include <stdio.h>#include <omp.h>
int main(int argc, char** argv) { int id = -1, numThreads = -1;
// #pragma omp parallel { id = omp_get_thread_num(); numThreads = omp_get_num_threads(); if ( id == 0 ) { // thread with ID 0 is master printf(”Greetings from the master, #%d of %d threads\n\n”, id, numThreads); } else { // threads with IDs > 0 are workers printf(”Greetings from a worker, #%d of %d threads\n\n”, id, numThreads); } } return 0;}
csinparallel.org
Sample Executions$ ./masterWorker // pragma omp parallel disabledGreetings from the master, #0 of 1 threads
$ ./masterWorker // pragma omp parallel enabledGreetings from a worker, #1 of 8 threadsGreetings from a worker, #2 of 8 threadsGreetings from a worker, #5 of 8 threadsGreetings from a worker, #3 of 8 threadsGreetings from a worker, #6 of 8 threadsGreetings from the master, #0 of 8 threadsGreetings from a worker, #4 of 8 threadsGreetings from a worker, #7 of 8 threads
csinparallel.org
EXEMPLARS
csinparallel.org
Motivation
• Everyone in CS needs PDC• Not everyone is naturally drawn to PDC topics
How shall we motivate every CS undergraduate to learn the PDC they
will need for their careers?
csinparallel.org
Motivation
• Everyone in CS needs PDC• Not everyone is naturally drawn to PDC topics
Proposal: Teach PDC concepts with compelling applications.• Some CS students draw by concepts and tech• Other CS students drawn by the applications
How shall we motivate every CS undergraduate to learn the PDC they
will need for their careers?
csinparallel.org
Exemplars
An exemplar is:• A representative applied problem
plus • multiple code solutions implemented in
various PDC technologies, with commentary
csinparallel.org
Exemplar A (from EAPF Practicum)
• Compute π via numerical integration• Implemented solutions– Serial– Shared memory (OpenMP, TBB, pthreads, Windows
Threads, go language)– Distributed computing (MPI)– Accelerators (CUDA, Array Building Blocks)
• Comments:– Flexible uses: demo, concepts, tech, compare– But not a compelling application
csinparallel.org
Exemplar B (from EAPF Practicum)
• Drug design
• Implemented solutions– Serial– Shared memory (OpenMP, boost threads, go lang)– Map-reduce framework (Hadoop)
csinparallel.org
Exemplar B (from EAPF Practicum)
• Comments– Compelling application– Molecular dynamics, docking algorithm – Substitute for docking algorithm to score ligands: (score is maximal match count)
• Relates to genetic alignment algorithm• Multiple ways to scale: # ligands, ligand length, # cores• Random strings with random lengths for variable
computational load per ligand
csinparallel.org
Exemplars + Patterns
• Exemplar implementations offer a rich opportunity for learning patterns
• Examples– π as area (among 8 PDC implementations):
• Data Decomposition, Geometric Decomposition; Parallel For Loop, Master-Worker, Strict Data Parallel, Distributed Array; SIMD, Thread Pool, Message Passing, Collective Communication, Mutual Exclusion
– Drug design (among 4 PDC implementations):• Map-Reduce; Data Decomposition; Parallel For Loop, Fork-
Join, BSP, Master-Worker, Task Queue, Shared Array, Shared Queue; Thread Pool, Message Passing, Mutual Exclusion
Drug designπ as area
csinparallel.org
Conclusion
• Patterns – a meaning for “parallel thinking,” best practice from industry
• Patternlets – minimalist, scalable, executable programs, each illustrating a particular pattern’s behavior
• Exemplars – motivation, hands-on/demo, teaching resource, opportunities for PDC
• These are naturally combined and ready for deployment