Programming Multi-Core Systems with OpenMP · Programming Multi-Core Systems with OpenMP Clemens Grelck University of Amsterdam UvA / SURFsara High Performance Computing and Big Data

Programming Multi-Core Systemswith OpenMP

Clemens Grelck

University of Amsterdam

UvA / SURFsaraHigh Performance Computing and Big Data

Clemens Grelck, University of Amsterdam Programming Multi-Core Systems with OpenMP

Programming Multi-Core Systems with OpenMP

Targeted Architectures

OpenMP at a Glance

Loop Parallelization

Scheduling

Outlook


Target Multi-core Systems

Small-scale general-purpose (x86) multicore processors:

I Intel / AMD commodity processors with 2, 4, 6 or 8 cores

I potentially hyperthreaded



Medium-scale server systems:

I multiple (2 or 4 in practice) identical processors

I each processor with several cores

I high bandwidth data path between processors



Large-scale shared address space compute systems:

I large number of slightly simpler cores

I SUN MicroSystems / Oracle Niagara / UltraSparc T series

I up to 512 hardware threads (T3-4 server)


Symmetric Multiprocessor Architecture Model

core core corecore corecore corecore core core corecorecore core corecore

L2

L3 cache

L2

L1

Shared Address Space

L3 cache L3 cache L3 cache

L1 L1DI

L2

L1 L1DI

L2

L1 L1DI

L2

L1 L1DI

L2

L1 L1DI

L2

L1 L1DI

L2

L1DI

L1 L1DI

L2

L1 L1DI

L2

L1 L1DI

L2

L1 L1DI

L2

L1 L1DI

L2

L1 L1DI

L2

L1 L1DI

L2

L1 L1DI

L2

L1 L1DI

Characteristics:

I Shared address space notion of shared memory

I Multiple levels of hardware-coherent caches

I Multiple processors

I Each processor has multiple cores

I Each core has multiple hardware threads




OpenMP at a Glance


Scheduling

Outlook


Design Rationale of OpenMP

Ideal:

I Automatic parallelisation of sequential code.

I No additional parallelisation e↵ort for development,debugging, maintenance, etc.

Problem:

I Data dependences are di�cult to assess.

I Compilers must be conservative in their assumptions.

Way out:

I Take or write ordinary sequential program.

I Add annotations/pragmas/compiler directives that guideparallelisation.

I Let the compiler generate the corresponding code.



Ideal:



Problem:



Way out:






Ideal:



Problem:



Way out:





OpenMP at a Glance

OpenMP as a programming interface:

I Compiler directivesI Library functionsI Environment variables

C/C++ version:

#pragma omp name [clause]*

structured block

Fortran version:

!$ OMP name [ clause [, clause]*]

code block

!$ OMP END name


OpenMP at a Glance

OpenMP as a programming interface:

I Compiler directivesI Library functionsI Environment variables

C/C++ version:

#pragma omp name [clause]*

structured block

Fortran version:

!$ OMP name [ clause [, clause]*]

code block

!$ OMP END name


Hello World with OpenMP

Example program:

#include "omp.h"

#include <stdio.h>

int main()

{

printf( "Starting execution with %d threads .\n",

omp_get_num_threads ());

printf( "Hello world says thread %d of %d.\n",

omp_get_thread_num (),


printf( "Execution of %d threads terminated .\n",


return 0;

}


Hello World with OpenMP

#include "omp.h"

#include <stdio.h>

int main()

{








return 0;

}

Compiling the code:

gcc -fopenmp hello_world.c


Hello World with OpenMP#include "omp.h"

#include <stdio.h>

int main()

{








return 0;

}

Running the compiled code:

Starting execution with 1 threads.

Hello world says thread 0 of 1.

Execution of 1 threads terminated.


Hello World with OpenMP — now in parallel

#include "omp.h"

#include <stdio.h>

int main()

{



#pragma omp parallel

{




}



return 0;

}



Running the code with 4 threads:









Running the code with 4 threads:







Who determines number of threads ?

I Environment variable: export OMP_NUM_THREADS=4

I Library function: void omp_set_num_threads( int)


OpenMP Execution Model

Classical Fork/Join:

Master and slave threads concurrently

Master thread executes serial code.

Master thread encounters parallel directive.

Implicit barrier, wait for all threads to finish.

Master thread resumes serial execution.

execute parallel block.




OpenMP at a Glance


Scheduling

Outlook


Simple Loop Parallelisation

Example: element-wise vector product:

void elem_prod( double *c, double *a, double *b, int len)

{

int i;

for (i=0; i<len; i++)

{

c[i] = a[i] * b[i];

}

}

Idea of parallelisation:

I Have each thread compute some disjoint part of the vectors:



Example: element-wise vector product:


{

int i;


{

c[i] = a[i] * b[i];

}

}

Idea of parallelisation:

I Have each thread compute some disjoint part of the vectors:

*= = = =

c

b

a

* * *



Example: parallelised element-wise vector product:


{

int i;

#pragma omp parallel for


{

c[i] = a[i] * b[i];

}

}

Prerequisite:

I No data dependence between any two iterations.

I Caution: YOU claim this property !!





{

int i;



{

c[i] = a[i] * b[i];

}

}

Prerequisite:







{

int i;



{

c[i] = a[i] * b[i];

}

}

Prerequisite:




Directive #pragma omp parallel for

What the compiler directive does for you:

I It starts additional worker threads depending onOMP NUM THREADS or omp set num threads().

I It divides the iteration space among all threads.

I It lets all threads execute loop restricted to their mutuallydisjoint subsets.

I It synchronizes all threads at an implicit barrier.

I It terminates the worker threads.

Restrictions:

I The directive must directly precede for-loop.

I The for-loop must match a constrained pattern.

I The trip-count of the for-loop must be known in advance.


Directive #pragma omp parallel for

What the compiler directive does for you:

I It starts additional worker threads depending onOMP NUM THREADS or omp set num threads().

I It divides the iteration space among all threads.

I It lets all threads execute loop restricted to their mutuallydisjoint subsets.

I It synchronizes all threads at an implicit barrier.

I It terminates the worker threads.

Restrictions:

I The directive must directly precede for-loop.

I The for-loop must match a constrained pattern.

I The trip-count of the for-loop must be known in advance.


Shared and Private Variables

Example:



{

c[i] = a[i] * b[i];

}

Private variables:

I One private instance for each threadI No communication between threads within parallel sectionI No communication between parallel and sequential sections

Shared variables:

I One shared instance for all threadsI Allows communication between threads within parallel sectionI Allows communication between parallel and sequential sections



Example:



{

c[i] = a[i] * b[i];

}

Who decides that a, b, c and len are shared variables,whereas i is private ??

Default rules:

I All variables are shared.

I Only loop variables of parallel loops are private.



Example:



{

c[i] = a[i] * b[i];

}

Who decides that a, b, c and len are shared variables,whereas i is private ??

Default rules:

I All variables are shared.

I Only loop variables of parallel loops are private.



The default rule is not what you want ?

I The shared clause determines shared variables

I The private clause determines private variables

Example with explicit clauses:

#pragma omp parallel for private( i) shared( c, a, b, len)


{

c[i] = a[i] * b[i];

}


From Vector Product to Scalar Product

Scalar product:double scalar_prod( double *a, double *b, int len)

{

int i; double ep , sp=0.0;


{

ep = a[i] * b[i];

sp = sp + ep;

}

return sp;

}

sp

a

* * * *b

ep

+

= = = =



sp

a

* * * *b

ep

+

= = = =

Reduction operations:

I Reductions reduce a set of elements to one

I Using some function

I Examples: sum or product of a set of numbers

Can we parallelise reductions ?



sp

a

* * * *b

ep

+

= = = =

Reduction operations:

I Reductions reduce a set of elements to one

I Using some function

I Examples: sum or product of a set of numbers

Can we parallelise reductions ?


Parallel Reduction


{



{

ep = a[i] * b[i];

sp = sp + ep;

}

return sp;

}

Properties:

I Reduction introduces a loop-carried dependence.

I Good news: operation is associativeI Order of reductions irrelevant for final resultI (If we ignore deficiencies of computer arithmetic...)

Private variables ?

Shared variables ?


Parallel Reduction


{



{

ep = a[i] * b[i];

sp = sp + ep;

}

return sp;

}

Properties:

I Reduction introduces a loop-carried dependence.I Good news: operation is associative

I Order of reductions irrelevant for final resultI (If we ignore deficiencies of computer arithmetic...)

Private variables ?

Shared variables ?


Parallel Reduction


{



{

ep = a[i] * b[i];

sp = sp + ep;

}

return sp;

}

Properties:

I Reduction introduces a loop-carried dependence.I Good news: operation is associativeI Order of reductions irrelevant for final result

I (If we ignore deficiencies of computer arithmetic...)

Private variables ?

Shared variables ?


Parallel Reduction


{



{

ep = a[i] * b[i];

sp = sp + ep;

}

return sp;

}

Properties:

I Reduction introduces a loop-carried dependence.I Good news: operation is associativeI Order of reductions irrelevant for final resultI (If we ignore deficiencies of computer arithmetic...)

Private variables ?

Shared variables ?


Parallel Reduction


{



{

ep = a[i] * b[i];

sp = sp + ep;

}

return sp;

}

Properties:

I Reduction introduces a loop-carried dependence.I Good news: operation is associativeI Order of reductions irrelevant for final resultI (If we ignore deficiencies of computer arithmetic...)

Private variables ?

Shared variables ?


Parallel Reduction

Parallel scalar product:

double scalar_prod( double *a, double *b, int len)

{


#pragma omp parallel for shared( a, b, len , sp) \

private( i, ep)


{

ep = a[i] * b[i];

sp = sp + ep;

}

return sp;

}

Is this really correct ?


Parallel Reduction

Parallel scalar product:


{



private( i, ep)


{

ep = a[i] * b[i];

sp = sp + ep;

}

return sp;

}

Is this really correct ?


A Look into Assembly

The troublemaker in C:

sp = sp + ep;

The troublemaker in (pseudo) assembly:

load sp -> reg1

load ep -> reg2

add reg1 , reg2 -> reg1

store reg1 -> sp

Problem:

I Same code executed by multiple threads simultaneously

I Any interleaving of assembly instructions possible


A Look into Assembly

The troublemaker in C:

sp = sp + ep;

The troublemaker in (pseudo) assembly:

load sp -> reg1

load ep -> reg2

add reg1 , reg2 -> reg1

store reg1 -> sp

Problem:

I Same code executed by multiple threads simultaneously

I Any interleaving of assembly instructions possible


Interleaved Execution of Threads

Thread 1:load sp -> reg1

load ep -> reg2

add reg1, reg2 -> reg1

store reg1 -> sp


load ep -> reg2


store reg1 -> sp

Thread 1 private Shared Thread 2 privateep reg1 reg2 sp ep reg1 reg28 0 0 29 5 0 0

8 29 0 29 5 0 08 29 8 29 5 0 08 37 8 29 5 0 08 37 8 37 5 0 08 37 8 37 5 37 08 37 8 37 5 37 58 37 8 37 5 42 58 37 8 42 5 42 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp

Thread 1 private Shared Thread 2 privateep reg1 reg2 sp ep reg1 reg28 0 0 29 5 0 08 29 0 29 5 0 0

8 29 8 29 5 0 08 37 8 29 5 0 08 37 8 37 5 0 08 37 8 37 5 37 08 37 8 37 5 37 58 37 8 37 5 42 58 37 8 42 5 42 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp

Thread 1 private Shared Thread 2 privateep reg1 reg2 sp ep reg1 reg28 0 0 29 5 0 08 29 0 29 5 0 08 29 8 29 5 0 0

8 37 8 29 5 0 08 37 8 37 5 0 08 37 8 37 5 37 08 37 8 37 5 37 58 37 8 37 5 42 58 37 8 42 5 42 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp

Thread 1 private Shared Thread 2 privateep reg1 reg2 sp ep reg1 reg28 0 0 29 5 0 08 29 0 29 5 0 08 29 8 29 5 0 08 37 8 29 5 0 0

8 37 8 37 5 0 08 37 8 37 5 37 08 37 8 37 5 37 58 37 8 37 5 42 58 37 8 42 5 42 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp

Thread 1 private Shared Thread 2 privateep reg1 reg2 sp ep reg1 reg28 0 0 29 5 0 08 29 0 29 5 0 08 29 8 29 5 0 08 37 8 29 5 0 08 37 8 37 5 0 0

8 37 8 37 5 37 08 37 8 37 5 37 58 37 8 37 5 42 58 37 8 42 5 42 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp

Thread 1 private Shared Thread 2 privateep reg1 reg2 sp ep reg1 reg28 0 0 29 5 0 08 29 0 29 5 0 08 29 8 29 5 0 08 37 8 29 5 0 08 37 8 37 5 0 08 37 8 37 5 37 0

8 37 8 37 5 37 58 37 8 37 5 42 58 37 8 42 5 42 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp

Thread 1 private Shared Thread 2 privateep reg1 reg2 sp ep reg1 reg28 0 0 29 5 0 08 29 0 29 5 0 08 29 8 29 5 0 08 37 8 29 5 0 08 37 8 37 5 0 08 37 8 37 5 37 08 37 8 37 5 37 5

8 37 8 37 5 42 58 37 8 42 5 42 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp

Thread 1 private Shared Thread 2 privateep reg1 reg2 sp ep reg1 reg28 0 0 29 5 0 08 29 0 29 5 0 08 29 8 29 5 0 08 37 8 29 5 0 08 37 8 37 5 0 08 37 8 37 5 37 08 37 8 37 5 37 58 37 8 37 5 42 5

8 37 8 42 5 42 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp

Thread 1 private Shared Thread 2 privateep reg1 reg2 sp ep reg1 reg28 0 0 29 5 0 08 29 0 29 5 0 08 29 8 29 5 0 08 37 8 29 5 0 08 37 8 37 5 0 08 37 8 37 5 37 08 37 8 37 5 37 58 37 8 37 5 42 58 37 8 42 5 42 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 0 0 29 5 29 08 0 0 29 5 29 58 0 0 29 5 34 58 0 0 34 5 34 58 34 0 34 5 34 58 34 8 34 5 34 58 42 8 34 5 34 58 42 8 42 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 0 0 29 5 29 58 0 0 29 5 34 58 0 0 34 5 34 58 34 0 34 5 34 58 34 8 34 5 34 58 42 8 34 5 34 58 42 8 42 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 0 0 29 5 34 58 0 0 34 5 34 58 34 0 34 5 34 58 34 8 34 5 34 58 42 8 34 5 34 58 42 8 42 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 0 0 34 5 34 58 34 0 34 5 34 58 34 8 34 5 34 58 42 8 34 5 34 58 42 8 42 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 34 0 34 5 34 58 34 8 34 5 34 58 42 8 34 5 34 58 42 8 42 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 34 8 34 5 34 58 42 8 34 5 34 58 42 8 42 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 42 8 34 5 34 58 42 8 42 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 42 8 42 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp





load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 29 0 29 5 0 08 29 0 29 5 29 08 29 8 29 5 29 08 29 8 29 5 29 58 37 8 29 5 29 58 37 8 29 5 34 58 37 8 37 5 34 58 37 8 34 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 29 0 29 5 29 08 29 8 29 5 29 08 29 8 29 5 29 58 37 8 29 5 29 58 37 8 29 5 34 58 37 8 37 5 34 58 37 8 34 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 29 8 29 5 29 08 29 8 29 5 29 58 37 8 29 5 29 58 37 8 29 5 34 58 37 8 37 5 34 58 37 8 34 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 29 8 29 5 29 58 37 8 29 5 29 58 37 8 29 5 34 58 37 8 37 5 34 58 37 8 34 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 37 8 29 5 29 58 37 8 29 5 34 58 37 8 37 5 34 58 37 8 34 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 37 8 29 5 34 58 37 8 37 5 34 58 37 8 34 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 37 8 37 5 34 58 37 8 34 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp


8 37 8 34 5 34 5




load ep -> reg2


store reg1 -> sp


load ep -> reg2


store reg1 -> sp



Race Condition / Data Race

Definition:

I A race condition / data race exists if the behaviour (themeaning) of a program depends on the execution order ofprogram parts (threads) whose temporal behaviour is beyondcontrol.

Origin of term:

I Electronics

I Two electric signals race against each other.

I Arrival order of input signals at gate determines output signal.

Big question: how can we avoid data races ?



Definition:


Origin of term:

I Electronics






Definition:


Origin of term:

I Electronics





How can we Avoid Data Races ?

Solution: critical sections

I Once a thread enters a critical section, it must leave it beforeany other thread can enter; no interleaving.

I Critical sections must be made explicit throughout program.

Example:double scalar_prod( double *a, double *b, int len)

{



private( i, ep)

for (i=0; i<len; i++) {

ep = a[i] * b[i];

#pragma omp critical

{

sp = sp + ep;

} }

return sp;

}


Critical Sections

Parallel scalar product:double scalar_prod( double *a, double *b, int len)

{



private( i, ep)

for (i=0; i<len; i++) {

ep = a[i] * b[i];

#pragma omp critical

{

sp = sp + ep;

} }

return sp;

}

The critical directive:

I Directive must immediately precede new statement block.I Statement block is executed without interleaving.I Directive implements critical section.


Named Critical Sections

Disadvantage:

I All critical sections in entire program are synchronised.

I Many might be unrelated.

I Unnecessary synchronisation overhead.

Solution: named critical directive

I Critical sections may be associated with names.

I Critical sections with identical names are synchronised.

I Critical sections with di↵erent names are executedconcurrently.


Named Critical Sections

Disadvantage:

I All critical sections in entire program are synchronised.

I Many might be unrelated.

I Unnecessary synchronisation overhead.

Solution: named critical directive

I Critical sections may be associated with names.

I Critical sections with identical names are synchronised.

I Critical sections with di↵erent names are executedconcurrently.


Named Critical Section

Scalar product:


{



private( i, ep)

for (i=0; i<len; i++) {

ep = a[i] * b[i];

#pragma omp critical (scalar_prod)

{

sp = sp + ep;

} }

return sp;

}

BUT: Is this really e�cient ?


Named Critical Section

Scalar product:


{



private( i, ep)

for (i=0; i<len; i++) {

ep = a[i] * b[i];

#pragma omp critical (scalar_prod)

{

sp = sp + ep;

} }

return sp;

}

BUT: Is this really e�cient ?


Reduction Clause

Scalar product:


{


#pragma omp parallel for shared( a, b, len) \

private( i, ep) \

reduction( +: sp)

for (i=0; i<len; i++) {

ep = a[i] * b[i];

sp = sp + ep;

}

return sp;

}


Reduction Clause

Scalar product:


{



private( i, ep) \

reduction( +: sp)

for (i=0; i<len; i++) {

ep = a[i] * b[i];

sp = sp + ep;

}

return sp;

}

a

* * * *b

ep

= = = =


Reduction Clause


{



private( i, ep) \

reduction( +: sp)

for (i=0; i<len; i++) {

ep = a[i] * b[i];

sp = sp + ep;

}

return sp;

}

Properties:

I Reduction clause only supports built-in reduction operations:+, *, ^, &, |, &&, ||, min, max.

I Bit accuracy not guaranteed.


Conditional Parallelisation

Problem:

I Parallel execution of aloop incurs overhead:

I creation of workerthreads

I schedulingI synchronisation barrier

I This overhead must beoutweighed by su�cientworkload.

I Workload depends onI loop body,I trip count.



Problem:

I Parallel execution of aloop incurs overhead:

I creation of workerthreads

I schedulingI synchronisation barrier

I This overhead must beoutweighed by su�cientworkload.

I Workload depends onI loop body,I trip count.

Example:

if (len < 1000) {


{

res[i] = a[i] * b[i];

}

}

else {



{

res[i] = a[i] * b[i];

}

}



Introducing the if-clause:

if (len < 1000) {

for (i=0; i<len; i++) {

res[i] = a[i] * b[i];

}

}

else {


for (i=0; i<len; i++) {

res[i] = a[i] * b[i];

}

}

#pragma omp parallel for if (len >= 1000)

for (i=0; i<len; i++) {

res[i] = a[i] * b[i];

}



If-clause:

#pragma omp parallel for if (len >= 1000)

for (i=0; i<len; i++) {

res[i] = a[i] * b[i];

}

Some facts:

I If-clause can contain any kind of C expression

I C expression may refer to all identifiers in scopeI C expression is evaluated first:

I false: sequential executionI true: parallel execution




OpenMP at a Glance


Scheduling

Outlook


Loop Scheduling

Definition:

I Loop scheduling determines which iterations are executed bywhich thread.

Aim:

I Equal workload distribution

timeworkwait

task 4

task 3

task 2

task 1

sy

nc

hro

niz

ati

on

ba

rrie

r


Loop Scheduling

Problem:

I Di↵erent situations require di↵erent techniques

The schedule clause:

#pragma omp parallel for schedule( <type > [, <chunk >])

for (...)

{

...

}

Properties:

I Clause selects one out of a set of scheduling techniques.

I Optionally, a chunk size can be specified.

I Default chunk size depends on scheduling technique.


Loop Scheduling

Static scheduling:

#pragma omp parallel for schedule( static)

I Loop is subdivided into as many chunks as threads exist.

I Often called block scheduling.

Illustration:


Loop Scheduling

Static scheduling with chunk size 1:

#pragma omp parallel for schedule( static , 1)

I Iterations are assigned to threads in a round-robin fashion.

I Also called cyclic scheduling.

Illustration:


Loop Scheduling

Static scheduling with chunk size ¡n¿:

#pragma omp parallel for schedule( static , <n>)

I Loop is subdivided into chunks of n iterations.

I Chunks are assigned to threads in a round-robin fashion.

I Also called block-cyclic scheduling.

Illustration:


Loop Scheduling

Dynamic scheduling:

#pragma omp parallel for schedule( dynamic , <n>)


I Chunks are dynamically assigned to threads on their demand.

I Also called self scheduling.

I Default chunk size: 1 iteration.

Properties to keep in mind:

I Allows for dynamic load distribution and adjustment.

I Requires additional synchronization.

I Generates more overhead than static scheduling.


Loop Scheduling

Dynamic scheduling:

#pragma omp parallel for schedule( dynamic , <n>)


I Chunks are dynamically assigned to threads on their demand.

I Also called self scheduling.

I Default chunk size: 1 iteration.

Properties to keep in mind:

I Allows for dynamic load distribution and adjustment.

I Requires additional synchronization.

I Generates more overhead than static scheduling.


Loop Scheduling

Dilemma of chunk size selection:

I Small chunk sizes mean good load balancing, but highsynchronisation overhead.

I Large chunk sizes reduce synchronisation overhead, but resultin poor load balancing.

Rationale of guided scheduling:

I In the beginning, large chunks keep synchronisation overheadsmall.

I When approaching the final barrier, small chunks balanceworkload.


Loop Scheduling

Dilemma of chunk size selection:

I Small chunk sizes mean good load balancing, but highsynchronisation overhead.

I Large chunk sizes reduce synchronisation overhead, but resultin poor load balancing.

Rationale of guided scheduling:

I In the beginning, large chunks keep synchronisation overheadsmall.

I When approaching the final barrier, small chunks balanceworkload.


Loop Scheduling

Guided scheduling:#pragma omp parallel for schedule( guided , <n>)

I Chunks are dynamically assigned to threads on their demand.I Initial chunk size is implementation dependent.I Chunk size decreases exponentially with every assignment.I Also called guided self scheduling.I Minimum chunk size: n (default: 1)

Example:

I Total number of iterations: 250I Initial / minimal chunk size: 50 / 5I Current chunk size: 80% of last chunk size:

50 – 40 – 32 – 26 – 21 – 17 – 14 – 12 – 10 – 8 – 6 – 5 – 5 – 4I All properties are implementation-dependent !


Loop Scheduling

Guided scheduling:#pragma omp parallel for schedule( guided , <n>)

I Chunks are dynamically assigned to threads on their demand.I Initial chunk size is implementation dependent.I Chunk size decreases exponentially with every assignment.I Also called guided self scheduling.I Minimum chunk size: n (default: 1)

Example:

I Total number of iterations: 250I Initial / minimal chunk size: 50 / 5I Current chunk size: 80% of last chunk size:

50 – 40 – 32 – 26 – 21 – 17 – 14 – 12 – 10 – 8 – 6 – 5 – 5 – 4I All properties are implementation-dependent !


Choice of Scheduling Technique

Which scheduling to choose when ?

I Depends on your code !I Crucial question: is the amount of computational work per

iteration (roughly) the same for each iteration or not?

Static scheduling techniques:

I Preferable for uniform workload distributionsI Minimal overheadI (Block-)Cyclic schedulings may be useful for regular uneven

workload distributionsI (Block-)Cyclic schedulings may run into cache issues

Dynamic scheduling techniques:

I Preferable for irregular workload distributionsI Additional synchronisation overhead needs compensationI Guided self-scheduling usually superior
























OpenMP at a Glance


Scheduling

Outlook


What’s More ?

More in OpenMP-2:

I Decouple parallel regions from work sharing

I Control synchronisation barriers

I Task parallel sections

I Low-level locks and condition variables

I ...

More in OpenMP-3:

I Nested parallel regions

I Spawning and synchronisation of tasks

I ...

More information:

I www.openmp.org


What’s More ?

More in OpenMP-2:





I ...

More in OpenMP-3:



I ...

More information:

I www.openmp.org


What’s More ?

More in OpenMP-2:





I ...

More in OpenMP-3:



I ...

More information:

I www.openmp.org


The End: Questions ?


Programming Multi-Core Systems with OpenMP · Programming Multi-Core Systems with OpenMP Clemens Grelck University of Amsterdam UvA / SURFsara High Performance Computing and Big Data

Documents