4.1 Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, @ 2004 Pearson Education Inc. All rights reserved. Chapter 4 Partitioning and Divide-and-Conquer Strategies • Two fundamental techniques in parallel programming - Partitioning - Divide-and-conquer • Typical problems solved with these techniques
37
Embed
4.1 Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, @
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
4.1Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Chapter 4Partitioning
and Divide-and-Conquer Strategies
• Two fundamental techniques in parallel programming- Partitioning- Divide-and-conquer
• Typical problems solved with these techniques
4.2Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Partitioning/Divide and Conquer Examples
Many possibilities.
• Operations on sequences of number such as simply adding them together
• Several sorting algorithms can often be partitioned or constructed in a recursive fashion
• Numerical integration
• N-body problem
4.3Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Partitioning
• Dividing the problem into parts.• Basis for all parallel programming, in one form or another.• Domain decomposition - partitioning applied to the
program data.• Dividing the data and operating upon the divided data
concurrently.• Functional decomposition - partitioning applied to the
functions of a program.• Dividing the program into independent functions and
executing the functions concurrently.• Domain decomposition forms the foundation of most
parallel algorithms
4.4Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Partitioning a sequence of numbers into parts and adding the parts
4.5Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Number Summation Revisited• Our previous implementation used broadcast() and reduce() to send
the data and combine the result• The following pseudocode uses individual send()s and receive()s:1. Master:
s = n/p;for (i=0,x=0;I<p;I++,x=x+s) send(&numbers[x],s,P[I]);sum = 0;for (i=0;i<p;i++){ recv(&part_sum,P[ANY]); sum = sum + part_sum;}
4.16Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Quadtree
4.17Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Dividing an image
4.18Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Example 1: Sorting Using Bucket Sort
• Most sequential sorting algorithms are based on comparing and exchanging pairs of numbers
• Bucket sort is not based on compare and exchange, but is naturally a partitioning method
• Works well only if the numbers to be sorted are uniformly distributed across a known interval
• If all the numbers in the interval 0 to a-1– Divide numbers into m regions 0 to a/m-1, a/m to 2a/m-1,…– One “bucket” is assigned to hold numbers that fall a region
4.19Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Bucket Sort
4.20Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Sequential Bucket Sort: Partitioning into Buckets
• Need to determine placement of each number into appropriate bucket
• Given a number r, compare it with the start of regions a/m, 2a/m, 3a/m, …– Can take m-1 comparisons to determine right bucket
• Better alternative: multiply r by m/a to determine its bucket– Requires one computational step, except that division can be
expensive• If m is a power of 2, bucket identification even faster
– If m = 2k, higher k bits of r determine r’s bucket• Assume that placing a number into a bucket requires one step
– placing all the numbers requires n steps.• If the numbers are uniformly distributed each bucket will contain
n/m numbers
4.21Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Sequential Bucket Sort: Sorting the Buckets
• After placing the numbers, each bucket needs to be sorted
• n numbers can be sorted sequentially in O(n log n) time• Best time for sequential compare and exchange sorting
• Thus, each bucket can be sorted in O(n/m log n/m) time, sequentially
• Assuming no concatenation overhead, the sequential time is
))/log(()/log())/log()/(( mnnOmnnnmnmnmnts
4.22Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Parallel Bucket Sort
• Can be parallelized by assigning one processor per bucket
• The computation time will now be (with m=p):
• This equation means that each processor examines each of the numbers – inefficient
• Can be improved by removing each examined number from the list at it is bucketed
)/log()/( pnpnntcomp
4.23Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Parallel version of bucket sortSimple approach
Assign one processor for each bucket.
4.24Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Further Parallelization
• Partition sequence into m regions, one region for each processor.
• Each processor maintains p “small” buckets and separates the numbers in its region into its own small buckets.
• Small buckets then emptied into p final buckets for sorting– Requires each processor to send one small bucket to each
of the other processors (bucket i to processor i).
• The algorithms has four phases:– Partitioning numbers– Placement into p small buckets– Sending small buckets to large buckets– Sorting large buckets
4.25Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Another parallel version of bucket sort
Introduces new message-passing operation - all-to-all broadcast.
4.26Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Analysis• Phase 1 - Computation and Communication
• the p partitions, containing n/p numbers each, are sent to the processes. Using a broadcast (or scatter) routine• tcomm1 = tstartup + ntdata
• Phase 2 – Computation• To separate each partition of n/p numbers into p small
buckets requires the time• tcomp2 = n/p
4.27Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Analysis• Phase 3 – Communication
• Next, the small buckets are distributed (no computation).• Each small bucket will have about n/p2 numbers.• Each process must send the contents of p-1 small
buckets to other processes• Since each process of the p processes must make this
communication, we havetcomm3 = p(p-1)(tstartup+(n/p2)tdata
• The above assumes the communications do not overlap in time and individual send()s are used
• If the communication could overlap (MPI_Alltoall()), we have:
tcomm3 = (p-1)(tstartup+(n/p2)tdata
4.28Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
4.29Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
“all-to-all” broadcast routineSends data from each process to every other process
4.30Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
“all-to-all” routine actually transfers rows of an array to columns:Transposes a matrix.
4.31Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Example 2: Numerical integration
• In many areas of mathematics and science it is necessary to calculate the area bounded by y = f(x), the axis, and the lines x = a, and x = b.
• The area can be calculated using definite integrals• Can be difficult or impossible sometimes
• Typically, approximations methods are used to calculate such area• Trapezoidal rule• Simpson rule• Adaptive quadrature methods
• Area approximated by partitioning the interval [a, b] into n equal pieces and inscribing a rectangle above each sub-interval thus obtained
4.32Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Numerical integration using rectangles.
Each region calculated using an approximation given by rectangles:Aligning the rectangles:
4.33Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Trapezoidal Rule:Static Assignment
• One process is assigned statically to compute each region, before the computation begins
• As before an SPMD programming model is used• Each calculation has same form
• With p processors to compute the area between x=a and x=b, each process will handle a region of size• (b-a)/p
• Each process will compute its area in the manner shown by the code overleaf
4.34Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Numerical integration using trapezoidal method
May not be better!
4.35Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Trapezoidal Rule (cont’d)if(i==master){
printf(“Enter # of intervals “);
scanf(“%d”,&n);
}
bcast(&n,p[group]);
region=(b-1)/p;
start = a + region*i;
end = start+region;
d =(b-a)/n;
area=0.0;
for(x=start; x<end; x=x+d)
area = area + 0.5*(f(x)+f(x+d))*d;
}
reduce_add(&integral,&area,p[group])
4.36Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Trapezoidal Rule: Optimization
• An implementation of this formula:
area = 0.5*(f(start)+f(end))*d;
for(x=start+d; x<end; x=x+d)
area = area + f(x);
area = area * d;
2
))())1(((...
2
))2()((
2
))()(( bfnafafafafafarea
4.37Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen,
@ 2004 Pearson Education Inc. All rights reserved.
Adaptive Quadrature
Solution adapts to shape of curve. Use three areas, A, B, and C.Computation terminated when largest of A and B sufficiently close to sum of remain two areas .