Creating Coarse- Grained Parallelism Chapter 6 of Allen and Kennedy Dan Guez
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Creating Coarse-Grained Parallelism
Chapter 6 of Allen and Kennedy
Dan Guez
Introduction Pervious lectures: Fine-Grained
Parallelism Superscalar and vector architecture Parallelizing Inner loops
This lecture: Coarse-Grained Parallelism Symmetric Multi Processor (SMP)
architecture Parallelizing Outer loops
SMP Architecture
Multiple asynchronous Processors Shared memory Synchronization required!
Communication between processors Barrier as synchronization mechanism Expensive!
Roadmap
Single Loop Methods Privatization Alignment Loop Fusion
Coarse-Grained vs. Fine Grained
Coarse-GrainedFine-Grained
PrivatizationScalar Expansion
AlignmentLoop Distribution
Loop fusion
Roadmap
Privatization Alignment Loop Fusion
Scalar Expansion - Reminder
Scalar Expansion – loop carried dependences elimination+ vectorizaion
DO I = 1, NT = A(I)A(I) = B(I)B(I) = T
ENDDO
DO I = 1, NT(I) = A(I)A(I) = B(I)B(I) = T(I)
ENDDO
T(1:N) = A(1:N)A(1:N) = B(1:N)B(1:N) = T(1:N)
Scalar
Expansion
Vectoring
Privatization Privatization - loop carried
dependences elimination
DO I = 1, NT = A(I)A(I) = B(I)B(I) = T
ENDDO
PARALLEL DO I = 1, NPRIVATE tt = A(I)A(I) = B(I)B(I) = t
END PARALLEL DO
Privatization
Privatization - Definition
A scalar variable x defined within a loop is said to be privatizable with respect to that loop if and only if every path from the beginning of the loop body to a use of x within the loop body must pass through a definition of x before reaching that use
Privatization – formal solution For each block x in the loop body
define equation (upward-exposed variables):
use(x) – the set of all variables used within block x that have no prior definitions within the block
def(x) – variables defined in x Solve the above equations…
Privatization – formal solution
The set of private variables is:
B – collection of loop body blocks b0 – the entry block to the loop body
Privatization - Theorem
A variable x defined in a loop may be made private if and only if the SSA graph for the variable does not have a Φ-node at the entry to the loop
Roadmap
Privatization Alignment Loop Fusion
Loop Distribution
Distributing loops using codegen
DO I = 1, 100DO J = 1, 100
A(I,J) = B(I,J) + C(I,J)D(I,J) = A(I,J-1) *2
ENDDOENDDO
codegenDO I = 1, 100
DO J = 1, 100A(I,J) = B(I,J) + C(I,J)
ENDDODO J = 1, 100
D(I,J) = A(I,J-1) *2ENDDO
ENDDO
Loop Distribution - vector architectures
DO I = 1, 100DO J = 1, 100
A(I,J) = B(I,J) + C(I,J)ENDDODO J = 1, 100
D(I,J) = A(I,J-1) *2ENDDO
ENDDO
DO I = 1, 100A(I,1:100) = B(I, 1:100) + C(I, 1:100)D(I, 1:100) = A(I,0:99) *2
ENDDO
Loop Distribution - SMP architectures
DO I = 1, 100DO J = 1, 100
A(I,J) = B(I,J) + C(I,J)ENDDOBarrier()DO J = 1, 100
D(I,J) = A(I,J-1) *2ENDDO
ENDDO
The solution - Alignment
DO I = 2, NS1 A(I) = B(I) + C(I)S2 D(I) = A(I-1) *2
ENDDO
S1
S2
I2 N
Basic Alignment
S1
S2
DO I = 1, NS1 IF (I > 1) A(I) = B(I) + C(I)S2 IF (I< N) D(I+1) = A(I) *2
ENDDO
DO I = 2, NS1 A(I) = B(I) + C(I)S2 D(I) = A(I-1) *2
ENDDO
I2 N1
Optimized Alignment – Option 1
S1
S2
DO I = 1, N-1J=I ; IF (I = 1) J = N A(J) = B(J) + C(J)D(I+1) = A(I) *2
ENDDO
DO I = 1, NS1 IF(I>1) A(I) = B(I) + C(I)S2 IF(I<N) D(I+1) = A(I) *2
ENDDO
I2 N1
Optimized Alignment – Option 2
S1
S2
D(2) = A(1)*2DO I = 2, N - 1
A(I) = B(I) + C(I)D(I+1) = A(I) *2
ENDDOA(N) = B(N)+C(N)
DO I = 1, NS1 IF(I>1) A(I) = B(I) + C(I)S2 IF(I<N) D(I+1) = A(I) *2
ENDDO
I2 N1
Alignment problems
Recurrence – impossible to align Different dependency distances –
Alignment fails (Alignment conflict)
DO I = 1, NA(I+1) = B(I) + C
X(I) = A(I+1) +A(I) ENDDO
DO I = 0, NIF (I>0) A(I+1) = B(I) + C
IF (I<N) X(I+1) = A(I+2) +A(I+1) ENDDO
Code Replication
Solves Alignment conflict
DO I = 1, NA(I+1) = B(I) + CX(I) = A(I+1)
+A(I)ENDDO
DO I = 1, NA(I+1) = B(I) + CIF (I=1)
t = A(I)ELSE
t = B(I-1) + CX(I) = A(I+1) + t
ENDDO
Alignment Graph
G = (V,E) – Directed Acyclic Graph. V - Set of loop body statements
Labeling o(v) – vertex offset E – Set of dependences
Labeling d(e) - dependence distance
Alignment Graph - Example
DO I = 1, N S1 A(I+2) = B(I) + CS2 X(I+1) = A(I) + DS3 Y(I) = A(I+1) +X(I)
ENDDO
S1
S2
S3
d=2d=1
d=1o = 0
o = 0
o = 0
Alignment Goal
The Graph G = (V,E) is said to be carry-free if for each edge e=(u,v)
o(u) + d(e) = o(v) Alignment procedure gets
alignment graph and generated carry-free alignment graph
Align Procedure
While V is not empty Add to worklist W arbitrary vertex v
from V While W is not empty
Remove vertex u from worklist W Align all adjacent vertices of u, replicate
node if different alignments required Add new aligned nodes in W
Align Procedure-Example
S1
S2
S3
d=2d=1
d=1o = 0
o = 0
o = 0
o = -1
o = -1
S1`
o = -3
d=2
GenAlign Procedure Set variables
hi maximal vertex offset lo minimal vertex offset Ivar original iteration variable Lvar original loop lower bound Uvar original loop upper bound
Generate loop statement “DO Ivar = Lvar-hi, Uvar + lo”
GenAlign Procedure cont. Scan vertices in a topological sort order
Let v be the current vertex if o(v) = lo then generate
“IF (Ivar >= Lvar-o(v) ) THEN “ + The related statement of v with Ivar+o(v)
substituted for Ivar else if o(v) = hi then generate
“IF (Ivar =< Uvar-o(v) ) THEN “… else generate
“IF (Ivar>=Lvar-o(v) AND Ivar<=Uvar-o(v)) THEN“…
GenAlign Procedure cont. if v is a replicated vertex, replace the
statement S with the following “THEN
tv = RHS(S) with Ivar+o(v) substituted for IvarELSE
tv = LHS(S) with Ivar+o(v) substituted for IvarENDIF”
Where tv is new unique scalar. Replace reference at the sink of every
dependence from v by tv
GenAlign - Example
DO I = 1, N S1 A(I+2) = B(I) + CS2 X(I+1) = A(I) + DS3 Y(I) = A(I+1) +X(I)
ENDDO DO I = 1, N+3 S1 IF (I>=4) A(I-1) = B(I-3) + CS1` IF (I>=2 AND I<=N+1) THEN
t = B(I-1) + CELSE
t = A(I+1)ENDIF
S2 IF (I>=2 AND I<=N+1) X(I)=A(I-1)+DS3 IF (I<=N) Y(I)=t+X(I)
ENDDO
Roadmap
Privatization Alignment Loop Fusion
Loop Fusion - Motivation
DO I = 1, NA(I) = B(I) + 1C(I) = A(I) + C(I-1)D(I) = A(I) + X
ENDDO
DO I = 1, NA(I) = B(I) + 1
ENDDODO I = 1, N
C(I) = A(I) + C(I-1)ENDDODO I = 1, N
D(I) = A(I) + XENDDO
Distribution
Parallelizable
Serial
Loop Fusion - Motivation
PARALLEL DO I = 1, NA(I) = B(I) + 1
ENDDODO I = 1, N
C(I) = A(I) + C(I-1)ENDDOPARALLE DO I = 1, N
D(I) = A(I) + XENDDO
Fusion
PARALLEL DO I = 1, NA(I) = B(I) + 1D(I) = A(I) + X
ENDDODO I = 1, N
C(I) = A(I) + C(I-1)ENDDO
Loop Fusion – Graphical View
L1
L2
L3
L1,
3
L2
Loop Fusion - Safety Constraints
PARALLEL DO I = 1, NA(I) = B(I) + 1
ENDDOPARALLE DO I = 1, N
D(I) = A(I+1) + XENDDO
FusionPARALLEL DO I = 1, N
A(I) = B(I) + 1
D(I) = A(I+1) + XENDDO
Fusion-preventing dependence constraint: the fused loops generates backward loop carried dependence
X
Loop Fusion - Safety Constraints
Ordering constraint: there is a path that contains a loop-independent dependence between the loops that can’t be fused with them
L1
L2
L3
L1,
3
L2X Fusion
Loop Fusion - Profitability Constraints
Separation constraint: do not fuse parallel loops with sequential loops…
Parallelism inhibiting constraint: the fused loop will have a forward carried dependence.PARALLEL DO I = 1, NA(I) = B(I) + 1
ENDDOPARALLE DO I = 1, N
D(I) = A(I-1) + XENDDO
FusionPARALLEL DO I = 1, N
A(I) = B(I) + 1
D(I) = A(I-1) + XENDDO
Typed Fusion - Definition
P=(G, T, m, B, t0) G=(V,E) directed acyclic graph
(dependence graph) T – set of types (parallel, sequential) m:VT mapping of types to vertices B – set of bad edges (constraints) t0 – objective type (parallel)
Main Data Structures
num[n] – holds the number of the node number of n in the fused graph
maxBadPrev[n] – holds the maximal vertex number of type t0 in the fused graph that cannot be fused with n“The maximal fused vertex that n is preventing from being further fused”
TypedFusion Example
1
3
4
6 5
2
1,
2,
3,
7
4,5,
0
1
1
33
update_successors(n) Set t = type(n) For each edge (n,m) do
If (t != t0) maxBadPrev[m] = MAX(maxBadPrev[m],
maxBadPrev[n])
Else If (type(m) != t0 or (n,m) in B) maxBadPrev[m] = MAX(maxBadPrev[m], num[n])
Else maxBadPrev[m] = MAX(maxBadPrev[m],
maxBadPrev[n])
TypedFusion Procedure
Scan the vertices in topological order For each vertex v
If v is of type t0
update_successors(v)fuse v with the first possible
available node Else
update_successors(v)create_new_node(v)
TypedFusion Example
1
4
5
7 8
6
3
2
TypedFusion Example
1
4
5
7 8
6
3
2
(0,1)
(0,1)
(0,2)
(1,3)
(1,4) (1,5)
(1,4)(4,6)
(maxBadPrev , num)
(0, ) (1, )
(0, )(1, ) (1, )
(4, ) (1, )
TypedFusion Example
1,3
4
5,8
7
6
2
(1) (2)
(3)
(4) (5)
(6)
Fusing sequential loops…
1,3
2,4,6
5,8
7
(1)
(3)
(4)
(6)
Ordered Fusion
More than 2 types of loops: Different loop headers
Setting the “right” order of types to fuse is NP-hard Define priorities to each type
Type conflict
1 2
3 4
Ordered Fusion
1 2
4 5
3
6
1
4
5
7 8
6
3
2
Cohort Fusion
Cohort Fusion
Allows running sequential loops with parallel loop
Settings: Single type Bad edges
Fusion-preventing edges Parallelism-inhibiting edges Edges mixing parallel and sequential loops
Cohort Fusion
Pros Minimal number of barriers
Cons Bad load balancing
Related Documents
