A Sophomoric Introduction to Shared- Memory Parallelism and Concurrency Lecture 3 Parallel Prefix, Pack, and Sorting Steve Wolfman, based on work by Dan Grossman This file is licensed under a Creative Commons Attribution 3.0 Unported License; see reativecommons.org/licenses/by/3.0/. The materials were developed by Steve Wolfman, Alan Hu, and Dan
A Sophomoric Introduction to Shared-Memory Parallelism and Concurrency Lecture 3 Parallel Prefix, Pack, and Sorting. Steve Wolfman , based on work by Dan Grossman. LICENSE : This file is licensed under a Creative Commons Attribution 3.0 Unported License; see - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
A Sophomoric Introduction to Shared-Memory Parallelism and Concurrency
Lecture 3 Parallel Prefix, Pack, and Sorting
Steve Wolfman, based on work by Dan Grossman
LICENSE: This file is licensed under a Creative Commons Attribution 3.0 Unported License; see http://creativecommons.org/licenses/by/3.0/. The materials were developed by Steve Wolfman, Alan Hu, and Dan Grossman.
2Sophomoric Parallelism and Concurrency, Lecture 3
Learning Goals
• Judge appropriate contexts for and apply the parallel map, parallel reduce, and parallel prefix computation patterns.
• And also… lots of practice using map, reduce, work, span, general asymptotic analysis, tree structures, sorting algorithms, and more!
3Sophomoric Parallelism and Concurrency, Lecture 3
Outline
Done:– Simple ways to use parallelism for counting, summing, finding– (Even though in practice getting speed-up may not be simple)– Analysis of running time and implications of Amdahl’s Law
Now: Clever ways to parallelize more than is intuitively possible– Parallel prefix– Parallel pack (AKA filter)– Parallel sorting
• quicksort (not in place) • mergesort
4Sophomoric Parallelism and Concurrency, Lecture 3
The prefix-sum problem
Given a list of integers as input, produce a list of integers as output where output[i] = input[0]+input[1]+…+input[i]
In practice, of course, we’d use a sequential cutoff!
15Sophomoric Parallelism and Concurrency, Lecture 3
Parallel prefix, generalized
Can we use parallel prefix to calculate the minimum of all elements to the left of i?
In general, what property do we need for the operation we use in a parallel prefix computation?
16Sophomoric Parallelism and Concurrency, Lecture 3
Outline
Done:– Simple ways to use parallelism for counting, summing, finding– (Even though in practice getting speed-up may not be simple)– Analysis of running time and implications of Amdahl’s Law
Now: Clever ways to parallelize more than is intuitively possible– Parallel prefix– Parallel pack (AKA filter)– Parallel sorting
• quicksort (not in place) • mergesort
17Sophomoric Parallelism and Concurrency, Lecture 3
Pack
AKA, filter
Given an array input, produce an array output containing only elements such that f(elt) is true
As usual, we can make lots of efficiency tweaks… with no asymptotic impact.
21Sophomoric Parallelism and Concurrency, Lecture 3
Outline
Done:– Simple ways to use parallelism for counting, summing, finding– (Even though in practice getting speed-up may not be simple)– Analysis of running time and implications of Amdahl’s Law
Now: Clever ways to parallelize more than is intuitively possible– Parallel prefix– Parallel pack (AKA filter)– Parallel sorting
• quicksort (not in place) • mergesort
22Sophomoric Parallelism and Concurrency, Lecture 3
Parallelizing QuicksortRecall quicksort was sequential, in-place, expected time O(n lg n)
Best / expected case work
1. Pick a pivot element O(1)2. Partition all the data into: O(n)
A. The elements less than the pivotB. The pivotC. The elements greater than the pivot
3. Recursively sort A and C 2T(n/2)How do we parallelize this?What span do we get?
T(n) =
23Sophomoric Parallelism and Concurrency, Lecture 3
How good is O(lg n) Parallelism?
Given an infinite number of processors, O(lg n) faster.So… sort 109 elements 30 times faster?! That’s not much
Can’t we do better? What’s causing the trouble?
(Would using O(n) space help?)
24Sophomoric Parallelism and Concurrency, Lecture 3
Parallelizing QuicksortRecall quicksort was sequential, in-place, expected time O(n lg n)
Best / expected case work
1. Pick a pivot element O(1)2. Partition all the data into: O(n)
A. The elements less than the pivotB. The pivotC. The elements greater than the pivot
3. Recursively sort A and C 2T(n/2)How do we parallelize this?What span do we get?
T(n) =
25Sophomoric Parallelism and Concurrency, Lecture 3
Analyzing T(n) = lg n + T(n/2)
Turns out our techniques from way back at the start of the term will work just fine for this:
T(n) = lg n + T(n/2) if n > 1= 1 otherwise
26Sophomoric Parallelism and Concurrency, Lecture 3
Parallel Quicksort Example• Step 1: pick pivot as median of three
8 1 4 9 0 3 5 2 7 6
• Steps 2a and 2c (combinable): pack less than, then pack greater than into a second array– Fancy parallel prefix to pull this off not shown
1 4 0 3 5 2
1 4 0 3 5 2 6 8 9 7
• Step 3: Two recursive sorts in parallel(can limit extra space to one array of size n, as in mergesort)
27Sophomoric Parallelism and Concurrency, Lecture 3
Outline
Done:– Simple ways to use parallelism for counting, summing, finding– (Even though in practice getting speed-up may not be simple)– Analysis of running time and implications of Amdahl’s Law
Now: Clever ways to parallelize more than is intuitively possible– Parallel prefix– Parallel pack (AKA filter)– Parallel sorting
• quicksort (not in place) • mergesort
28Sophomoric Parallelism and Concurrency, Lecture 3
mergesort
Recall mergesort: sequential, not-in-place, worst-case O(n lg n)
1. Sort left half and right half 2T(n/2)2. Merge results O(n)
Just like quicksort, doing the two recursive sorts in parallel changes the recurrence for the span to T(n) = O(n) + 1T(n/2) O(n)• Again, parallelism is O(lg n)• To do better, need to parallelize the merge
– The trick won’t use parallel prefix this time
29Sophomoric Parallelism and Concurrency, Lecture 3
Parallelizing the merge
Need to merge two sorted subarrays (may not have the same size)
0 1 4 8 9 2 3 5 6 7
Idea: Suppose the larger subarray has n elements. In parallel:• merge the first n/2 elements of the larger half with the
“appropriate” elements of the smaller half• merge the second n/2 elements of the larger half with the
rest of the smaller half
30Sophomoric Parallelism and Concurrency, Lecture 3
Parallelizing the merge
0 4 6 8 9 1 2 3 5 7
31Sophomoric Parallelism and Concurrency, Lecture 3
Parallelizing the merge
0 4 6 8 9 1 2 3 5 7
1. Get median of bigger half: O(1) to compute middle index
32Sophomoric Parallelism and Concurrency, Lecture 3
Parallelizing the merge
0 4 6 8 9 1 2 3 5 7
1. Get median of bigger half: O(1) to compute middle index2. Find how to split the smaller half at the same value as the left-
half split: O(lg n) to do binary search on the sorted small half
33Sophomoric Parallelism and Concurrency, Lecture 3
Parallelizing the merge
0 4 6 8 9 1 2 3 5 7
1. Get median of bigger half: O(1) to compute middle index2. Find how to split the smaller half at the same value as the left-
half split: O(lg n) to do binary search on the sorted small half3. Size of two sub-merges conceptually splits output array: O(1)
34Sophomoric Parallelism and Concurrency, Lecture 3
Parallelizing the merge
0 4 6 8 9 1 2 3 5 7
1. Get median of bigger half: O(1) to compute middle index2. Find how to split the smaller half at the same value as the left-
half split: O(lg n) to do binary search on the sorted small half3. Size of two sub-merges conceptually splits output array: O(1)4. Do two submerges in parallel
0 1 2 3 4 5 6 7 8 9lo hi
35Sophomoric Parallelism and Concurrency, Lecture 3
The Recursion
0 4 6 8 9 1 2 3 5 7
0 4 1 2 3 5
When we do each merge in parallel, we split the bigger one in half and use binary search to split the smaller one
76 8 9
36Sophomoric Parallelism and Concurrency, Lecture 3
Analysis
• Sequential recurrence for mergesort:T(n) = 2T(n/2) + O(n) which is O(nlgn)
• Doing the two recursive calls in parallel but a sequential merge:work: same as sequential span: T(n)=1T(n/2)+O(n) which is O(n)
• Parallel merge makes work and span harder to compute– Each merge step does an extra O(lg n) binary search to find
how to split the smaller subarray– To merge n elements total, do two smaller merges of possibly
different sizes– But worst-case split is (1/4)n and (3/4)n
• When subarrays same size and “smaller” splits “all” / “none”
37Sophomoric Parallelism and Concurrency, Lecture 3
Analysis continued
For just a parallel merge of n elements:• Span is T(n) = T(3n/4) + O(lg n), which is O(lg2 n)• Work is T(n) = T(3n/4) + T(n/4) + O(lg n) which is O(n)• (neither bound is immediately obvious, but “trust me”)
So for mergesort with parallel merge overall:• Span is T(n) = 1T(n/2) + O(lg2 n), which is O(lg3 n)• Work is T(n) = 2T(n/2) + O(n), which is O(n lg n)
So parallelism (work / span) is O(n / lg2 n)– Not quite as good as quicksort, but worst-case guarantee– And as always this is just the asymptotic result
Looking for Answers?
38Sophomoric Parallelism and Concurrency, Lecture 3
39Sophomoric Parallelism and Concurrency, Lecture 3
The prefix-sum problem
Given a list of integers as input, produce a list of integers as output where output[i] = input[0]+input[1]+…+input[i]
In practice, of course, we’d use a sequential cutoff!
43Sophomoric Parallelism and Concurrency, Lecture 3
Parallel prefix, generalized
Can we use parallel prefix to calculate the minimum of all elements to the left of i?
Certainly! Just replace “sum” with “min” in step 1 of prefix and replace fromLeft with a fromLeft that tracks the smallest element left of this node’s range.
In general, what property do we need for the operation we use in a parallel prefix computation?
ASSOCIATIVITY! (And not commutativity, as it happens.)
44Sophomoric Parallelism and Concurrency, Lecture 3
As usual, we can make lots of efficiency tweaks… with no asymptotic impact.
45Sophomoric Parallelism and Concurrency, Lecture 3
Parallelizing QuicksortRecall quicksort was sequential, in-place, expected time O(n lg n)
Best / expected case work
1. Pick a pivot element O(1)2. Partition all the data into: O(n)
A. The elements less than the pivotB. The pivotC. The elements greater than the pivot
3. Recursively sort A and C 2T(n/2)
How should we parallelize this?Parallelize the recursive calls as we usually do in fork/join D&C.Parallelize the partition by doing two packs (filters) instead.
46Sophomoric Parallelism and Concurrency, Lecture 3
Parallelizing QuicksortRecall quicksort was sequential, in-place, expected time O(n lg n)
Best / expected case work
1. Pick a pivot element O(1)2. Partition all the data into: O(n)
A. The elements less than the pivotB. The pivotC. The elements greater than the pivot
3. Recursively sort A and C 2T(n/2)How do we parallelize this? First pass: parallel recursive calls in step 3.What span do we get?
T(n) = n + T (n/2) = n + n/2 + T (n/4) = n/1 + n/2 + n/4 + n/8 + … + 1 Θ(n)
(We replace the O(n) term in O(n) + T (n/2) with just n for simplicity of analysis.)
47Sophomoric Parallelism and Concurrency, Lecture 3
Analyzing T(n) = lg n + T(n/2)
Turns out our techniques from way back at the start of the term will work just fine for this:
T(n) = lg n + T(n/2) if n > 1= 1 otherwise
We get a sum like: lg n + (lg n) - 1 + (lg n) - 2 + (lg n) - 3 + … 3 + 2 + 1
Let’s replace lg n by x: x + x-1 + x-2 + x-3 + … 3 + 2 + 1