Parallel and Distributed Systems

Institute for Software Science – University of Vienna

P.Brezany

Parallel and Distributed Systems

Peter Brezany

Institute for Software Science

University of Vienna


P.Brezany2

Typical One-Processor Architecture (SISD

Architecture)

SISD : Single Instruction stream Single Data stream


P.Brezany3

Array Processor (SIMD Architecture)

SIMD: Single Instructionstream Multiple Data streams


P.Brezany4

Loop Parallelizing for SIMDSsA typical scientific program spends approx. 90% of its execution

timein loops.Example in Java:

float A[1000], B[1000]; for (int i = 1; i < 1000; i++) { A[i-1] = B[i]; }

The above loop can be expressed in Fortran 95 in the following way:

A(0:998) = B(1:999)

This statement can be directly mapped onto a SIMD processor.

There is an initiative to extend Java by similar constructs.


P.Brezany5

Parallel Multi-Processor-Hardware

(MIMD Architectures)• Distributed-memory machines (DM Multiprocessors, DM MIMDS)– Each processor has local memory and disk– Communication via message-passing– Hard to program: explicit data distribution– Goal: minimize communication

• Shared-memory machines (SM Multiprocessors, SM MIMDs, SMPs)– Shared global address space and disk– Communication via shared memory variables– Ease of programming– Goal: maximize locality, minimize false sharing

• Current trend: Cluster of SMPs


P.Brezany6

Distributed Memory Architecture

(Shared Nothing)

LocalMemory

LocalMemory

LocalMemory

LocalMemory

CPU CPU

Interconnection Network

CPUCPU


P.Brezany7

DMM: Shared Disk Architecture

LocalMemory

LocalMemory

LocalMemory

LocalMemory

CPU CPU CPU CPU


Global Shared Disk Subsystem


P.Brezany8

Loop Parallelizing for DM MIMDs

Example in Java:

float A[10], B[10];

for (int i = 1; i < 10; i++) { A[i] = B[i-1]; }

For two processors, P1 and P2, a straightforward solution would be:


P.Brezany9

Loop Parallelizing for DM MIMDs (2)


P.Brezany10

Loop Parallelizing for DM MIMDs (3)Code on P1: float A[5], B[5]; float temp; for (int i = 1; i < 6; i++) { if ( i == 5 ) { receive message from P2 into temp; A[i-1] = temp; { else A[i-1] = B[i]; } Code on P2: float A[5], B[5]; float temp; for (int i = 0; i < 5; i++) { if ( i == 0 ) { temp = B[0]; send temp to P1; { else A[i-1] = B[i]; }


P.Brezany11

Shared Memory Architecture(Shared Everything, SMP)

CPU CPU


CPUCPU

Global Shared Memory


P.Brezany12

Loop Parallelizing for SMPsExample in Java:

float A[1000], B[1000]; for (int i = 1; i < 1000; i++) { A[i-1] = B[i]; }

If we have, e.g. two processors, P1 and P2, a straightforward

(non-optimal) solution would be:

Code on P1: for (int i = 1; i < 500; i++) { A[i-1] = B[i]; } Code on P2: for (int i = 500; i < 1000; i++) { A[i-1] = B[i]; }Data elements of A and B are stored in the shared memory.


P.Brezany13

Cluster of SMPs

CPU


CPU

CPU CPU

4-CPUSMP

CPU CPU

CPU CPU

4-CPUSMP

CPU CPU

CPU CPU

4-CPUSMP

CPU CPU

CPU CPU

4-CPUSMP


P.Brezany14


P.Brezany15

Abstract Maschine Model


P.Brezany16

Cluster von PCs


P.Brezany17


P.Brezany18

Pipeline

No Pipeline


P.Brezany19

Towards Parallel Databases


P.Brezany20

Relational Data Model

Example

Cities Name Population LandMunich 1211617 BayernBremen 535058 Bremen. . . . . . . . .

Schema: (Cities: STRING, Population: INTEGER, Land: STRING

Relation (represented by a table): {(Munich, 1.211.617, Bayern), (Bremen, 535.058, Bremen), ...}

Key : {Name}


P.Brezany21

Relational Data Model - Queries

SELECT <Attribute List>

FROM <Relation Name>

[WHERE <Condition>] – option

.......... other options

Example: This is equivalent to:

SELECT * SELECT Name, Population, LandFROM Cities FROM Cities

SELECT Name, LandFROM CitiesWHERE Population > 600000


P.Brezany22


P.Brezany23


P.Brezany24

Grid Idea

Parallel and Distributed Systems

Documents