Top Banner
© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Matthew J. Sottile Timothy G. Mattson Timothy G. Mattson Craig E Rasmussen Craig E Rasmussen
37

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Dec 15, 2015

Download

Documents

Aubrie Hindson
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1

Concurrency in Programming Languages

Matthew J. SottileMatthew J. Sottile

Timothy G. MattsonTimothy G. Mattson

Craig E RasmussenCraig E Rasmussen

Page 2: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 2

Chapter 6 Objectives

• Examine the historical development of concurrent and parallel systems.– Hardware– Languages

• Discuss relationship that languages had with evolving hardware.

Page 3: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Outline

• Evolution of machines• Evolution of languages• Limits of automatic parallelization

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 3

Page 4: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Early concurrency

• Biggest challenge with early computers was keeping them busy.– Increasing utilization.

• Idle CPUs meant wasted cycles.

• Common source of idle CPUs: I/O

• How to use the CPU when a program is waiting on I/O?

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 4

Page 5: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Addressing the I/O problem

• Two early methods became popular:

– Multiprogramming– Interrupt driven I/O

• Concurrency addressed the utilization problem by allowing different programs to use resources when one becomes blocked on I/O.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 5

Page 6: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Multiprogramming

• Transparently multiplex the computing hardware to give the appearance of simultaneous execution of multiple programs.

• Prior to multicore, single CPU machines used this approach to provide multitasking.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 6

Page 7: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Interrupt Driven I/O

• Another development was interrupt driven I/O.

• This allows hardware to notify the CPU when I/O operations were complete.– Avoids inefficient active polling and checking by CPU.– CPU can do other work and only worry about I/O

operations when the I/O hardware tells it that they are ready.

• Interrupt-based hardware helps manage parallel devices within a machine.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 7

Page 8: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

CPU sophistication

• Multiprogramming and interrupt-based hardware addressed early utilization problems when CPUs were simple.

• CPUs continued to advance though:– Huge increases in speed (cycles/second)– More sophisticated instruction sets

• This led to more hardware advances to support the increased capabilities of newer CPUs.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 8

Page 9: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Memory performance

• The time difference between CPU clock cycles and physical I/O devices (like tapes) always were large.

• Soon CPUs overtook digital memory performance as well.– Memory itself looked slower and slower relative to the

CPU.– Results in the same utilization problem faced in the

early days with I/O devices.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 9

Page 10: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Hierarchical memory

• One approach to dealing with this growing gap between CPU performance and memory performance was caches.

• Place one or more small, fast memories between the CPU and main memory.– Recently accessed data is replicated there.– Locality assumption is often safe: memory that was

recently accessed is likely to be accessed again.– Caches speed up these subsequent accesses, and we

amortize cost to first access it from slow memory with multiple subsequence accesses from fast cache.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 10

Page 11: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Pipelining

• Another advance was to decompose instructions into pieces such that the CPU be structured like a factory assembly line.– Instructions start at one end, and as they pass through,

subtasks are performed such that at the end, the instruction is complete.

• This allows multiple instructions to be executing at any point in time, each at a different point in the pipeline.– This is a form of parallelism: instruction level parallelism.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 11

Page 12: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Pipelining

• Pipelining allowed for more complex instructions to be provided that required multiple clock cycles to complete.– Each clock cycle, part of the instruction could proceed.

• Instruction level parallelism allowed this multi-cycle complexity to be hidden.– If the pipeline could be kept full, then every cycle an

instruction could complete.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 12

Page 13: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Vector processing

• Another method for achieving parallelism in the CPU was to allow each instruction to operate on a set of data elements simultaneously.

• Vector processing tool this approach – instructions operated on small vectors of data elements.

• This became very popular in scientific computing, and later in multimedia and graphics processing.– Today’s graphics processing units are a modern

descendant of early vector machines.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 13

Page 14: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Dataflow

• In the 1970s and 1980s, dataflow was proposed an alternative architecture to the traditional designs dating back to the 1940s.

• In dataflow, programs are represented as graphs in which vertices represent computations and edges represent data flowing into and out of computations.– Large opportunities for parallelism – any computation

with data ready could execute.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 14

Page 15: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Dataflow

• Example: an expression (a+b)*(c+d) could be represented as the flow graph below.

• The independence of the + nodes means they can execute in parallel.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 15

Page 16: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Massively parallel machines

• In the 1980s and early 1990s, “massively parallel” machines with many, many parallel processors were created.

• Design goal: use many, many simple CPUs to solve problems with lots of available parallelism.– Many instructions would complete per second due to

high processor count.– This would outperform more systems with few complex

CPUs.– Relied on finding problems with lots of parallelism.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 16

Page 17: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Distributed systems and clusters

• Special purpose supercomputers (such as MPPs) were very expensive.

• Networks of workstations could achieve similar performance by writing programs that used message passing as the method for coordinating parallel processing elements.

• Performance was often lower than special purpose supercomputers due to network latency, but extreme cost savings to buy clusters of workstations outweighed performance impact.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 17

Page 18: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Today: Multicore and accelerators

• Processor manufacturers today can put multiple CPUs on a single physical chip.– Expensive shared memory multiprocessors of the past

are now cheap desktop or laptop processors!

• Demands of multimedia (video, 3D games) have led to adoption of multicore and vector processing in special purpose accelerators.– Most machines today have a graphics card that is more

capable than a vector supercomputer 20 to 30 years ago.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 18

Page 19: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Outline

• Evolution of machines• Evolution of languages• Limits of automatic parallelization

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 19

Page 20: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

FORTRAN

• The first widely used language was FORTRAN – the FORmula TRANslation language.– Dominant in numerical applications, which were the

common application area for early computers.

• Standarad FORTRAN took many decades to adopt concurrency constructs.

• Dialects of FORTRAN built for specific machines did adopt these constructs.– E.g.: IVTRAN for the ILLIAC IV added the “DO FOR

ALL” loop to provide parallel loops.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 20

Page 21: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

ALGOL

• The ALGOrithmic Language was introduced around the same time as FORTRAN.– Introduced control flow constructs present in modern

languages.

• ALGOL 68 introduced concurrency constructs.– Collateral clauses allowed the programmer to express

sequences of operations that could be executed in arbitrary order (or in parallel).

– Added a data type for semaphores used for synchronization purposes.

– Introduced “par begin”, a way of expressing that a block of statements can be executed in parallel.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 21

Page 22: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Concurrent Pascal and Modula

• Concurrent Pascal is a dialect of the Pascal language designed for operating systems software developers.– Operating systems are fundamentally concerned with

concurrency since the introduction of multiprogramming and parallel I/O devices.

• Added constructs representing processes and monitors.– Monitors are similar to objects in that they provide data

encapsulation and synchronization primitives for defining and enforcing critical sections that operate on this data.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 22

Page 23: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Communicating Sequential Processes

• CSP was a formal language defined to represent concurrent processes that interact with each other via message passing.

• The occam language was heavily influenced by CSP and provided language constructs for:– Creating processes.– Defining sequential and parallel sequences of

operations.– Channels for communication between processes.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 23

Page 24: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Ada

• Ada was a language designed for the US Dept. of Defense in the 1970s and 1980s, still in use today.

• Intended to be used for critical systems in which high software assurance was required.

• Included constructs for concurrent programming early in its development.

• Ada used the “task” abstraction to represent concurrently executing activities.– Communication via message passing or shared

memory.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 24

Page 25: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Ada

• Tasks communication is via rendezvous. • Tasks reach points where they block until another

task reaches a paired point, at which time they communicate and continue.– The tasks “rendezvous” with each other.

• Ada 95 also introduced protected objects for synchronization of data access.– Provides mutual exclusion primitives to the programmer.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 25

Page 26: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Functional languages

• Functional languages have been around since LISP in the 1950s.

• Typically considered to be at a higher level of abstraction from imperative languages like C.– E.g.: Mapping a function over a list. A functional

programmer doesn’t implement lower level details like how the list is represented or how the loop iterating over its elements is structured.

• Higher level of abstraction leaves more decision making up to the compiler.– Such as parallelization.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 26

Page 27: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Functional languages

• MultiLISP was a dialect of Scheme that focused on concurrent and parallel programming.

• Included the notion of a future variable.– A future is a variable that can be passed around but

may not have a value associated with it until a later time.

– Operations on these future values can be synchronized if they are required before they are available.

• Futures have influenced modern languages, like Java and X10.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 27

Page 28: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Dataflow languages

• Dataflow languages like Id and VAL were created to program dataflow hardware.– Purely functional core languages restricted side effects

and made derivation of data flow graphs easier.– Lack of side effects facilitates parallelism.– I-structures and M-structures were part of the Id

language to provide facilities for synchronization, memory side effects, and I/O.

• Modern languages like Haskell provide constructs (e.g.: Haskell MVars) that are based on features of dataflow languages.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 28

Page 29: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Logic languages

• Based on formal logic expressions.– Programmers stated sets of relations and facts to

encode their problems.– Fundamental core of the languages are logical

operators (AND, OR, NOT).– High degree of parallelism in logic expressions. If we

have an expression “A and B and C”, A, B, and C can be evaluated in parallel.

• Logic languages like PROLOG influenced modern languages like Erlang.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 29

Page 30: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Parallel languages

• A number of languages were explored that specifically focused on parallelism.– Earlier examples were focused on general purpose

programming, with concurrency constructs as a secondary concern.

• Languages like High Performance Fortran and ZPL instead focused on abstractions that were specifically designed to build parallel programs.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 30

Page 31: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Parallel languages

• In both HPF and ZPL, data distribution was a key concern.– Given a set of parallel processing elements, how does

the programmer describe how large logical data structures are physically decomposed across the processors?

– Goal was to let the compiler generate the often tedious and error prone code to handle data distribution and movement.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 31

Page 32: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Parallel languages

• Encourages users to take a “global view” of programs, not focus on local processor view.

• This lets programmer focus on the problem they want to solve, instead of details about how to map their problem onto a parallel machine.

• Parallel languages also focus on retargetability.– If parallelization decisions are fully controlled by the

compiler, then it can make different decisions for different platforms. Portability is easier in this case.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 32

Page 33: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Modern languages

• With the introduction of multicore, concurrency constructs are being added to most mainstream languages now.– Java: Threading and synchronization primitives.– Fortran: Co-Array Fortran added to 2008 standard.– .NET languages: Synchronization and threading– Clojure: LISP derivative with software transactional

memory– Scala: Concurrent functional language.– Haskell: Software transactional memory, Mvars,

threads.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 33

Page 34: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Modern languages

• Most new language features in concurrent languages are based on features explored in earlier languages.

• Studying older languages that include concurrency constructs is informative in understanding what motivated their design and creation.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 34

Page 35: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Outline

• Evolution of machines• Evolution of languages• Limits of automatic parallelization

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 35

Page 36: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Inferrence of parallelism is hard

• Most people agree that it is exceptionally difficult to automatically parallelize programs written in sequential languages.– Language features get in the way. E.g., pointers

introduce potential for aliasing, which restricts compiler freedom to parallelize.

– High-level abstractions are lost in low-level implementations. Complex loops and pointer-based data structures make it very challenging to infer structures that can be parallelized.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 36

Page 37: © 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 1 Concurrency in Programming Languages Matthew J. Sottile Timothy G. Mattson Craig.

Move towards concurrent languages

• Vectorizing and parallelizing compilers are very powerful, but they are reaching their limits as parallelism seen in practice increases.

• The big trend in language design is to introduce language features that are built to support concurrent and parallel programming.

© 2009 Matthew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen 37