1 Communication Inter-process Communication is the heart of a Distributed System Basic mechanisms are: RPC RMI MOM Streams.

1

Communication

Inter-process Communication is the heart of a Distributed System

Basic mechanisms are:•RPC•RMI•MOM•Streams

2

How to Communicate?

• Problem: a process in machine A wants to communicate (an item) to a process in machine B

How is this done?

• Build a message and send it through the network (cable) to the “other” side.

MANY things have to be agreed upon!1. Voltage (0/1)? For bit representation..2. Correctness of messages3. Representation of numbers, strings, and data items(in different

machines)4. Etc.

3

Approach: OSI Layered Protocols

OSI Open System Interconnection Reference:Layers, interfaces, and protocols in the OSI model.

2-1

4

Layered Protocols: as you dip into the layers you “add” more info

A typical message as it appears on the network.

2-2

5

OSI ModelDesign for “Open System”: allows the communication

of one system with another – possibly different one.

Open Systems are not exactly “Open” ; They are governed by a set of rules/regulations…

• Protocols: formalization of these rules

• Connection-oriented protocols

• Connectionless protocols.

6

Physical LevelMostly concerned with the transmission of 0/1s..

• Voltage level

• Rate of transmission

• Network connectors (plugs)

Example:

RS232c (standard for serial communication).

7

Data Link Layer: frames & checksums

Discussion between a receiver and a sender in the data link layer.

2-3

8

Network Layer• Does the “routing” of packets..

• Problem: shortest path is not always available!!

• Even if it is available “routing” may opt for a different path? Why?

• IP Internet Protocol (main representative of Network Layer)

• An IP packet (frame) can be sent without any setup.

• Virtual Path is gaining popularity (in ATM networks)

9

Transport LayerData will have to delivered to the other side

with no loss..

The job of the TL is exactly this!

Examples..

• TCP (Transmission Control protocol)

• UDP (Universal Datagram Protocol): almost similar to IP.

• RTP (Real-Time Protocol)

10

Example: Client-Server TCP

a) Normal operation of TCP.b) Transactional TCP (TCP for Transactions Suite)

2-4

11

High(-er) Level Protocols.. • Session and Presentation Layer

– Provides dialog facilities, keeps track of who is talking at this time, and provides synchronization if needed.

– All these are useful into long transfers as precautions can be taken for the transmission of data.

• Examples:– FTP (file transfer protocol)– HTTP (HyperText Transfer Protocol)– TELNET, SSH

12

Example of Higher Level Protocols: Middleware Protocols

An adapted reference model for networked communication that may enforce the notion of “atomicity”

2-5

13

Communication of Processes-RPCs

[Birell & Nelson TOCS94]• When a process A in machine a calls a routine on

machine B, the calling process A suspends and the execution of the called program in B takes place transparently..

• Information is transported from caller to callee in the parameters and the result returns

• No message passing is visible to the user.• Remote Procedure Call (RPC).• Idea: make a remote procedure call appear as a local one!

14

Conventional Procedure Call

a) Parameter passing in a local procedure call: the stack before the call to read (from main() )b) The stack while the called procedure is active

15

Client and Server Stubs• The client stub works in a similar way to a usual

system call (for instance read()). • The difference is that a different version (the client

stub) is loaded into the frame stack.• This stub does not ask the OS to get any data on

behalf of the process.• The stub packs the parameters into a single message

and requests the message to be sent to the server.• When the msg arrives into the server, the OS of the

server surrenders the msg to the server stub (server side equivalent).

16

Client and Server Stubs

Principle of RPC between a client and server program.

17

Steps of a Remote Procedure Call1. Client procedure calls client stub in normal way2. Client stub builds message, calls local OS3. Client's OS sends message to remote OS4. Remote OS gives message to server stub5. Server stub unpacks parameters, calls server6. Server does work, returns result to the stub7. Server stub packs it in message, calls local OS8. Server's OS sends message to client's OS9. Client's OS gives message to client stub10. Stub unpacks result, returns to client

18

Possible Issues

Passing parameters is always a problem..

• Pass by value

• Pass by reference

• Call by copy/restore

Marshaling of parameters has to initially take place.

19

Passing Value Parameters

Steps involved in doing remote computation through RPC

2-8

20

Possible Problems with Passing Value Parameters

a) Original message on the Pentium (little Endian-number their bytes from right to left)

b) The message after receipt on the SPARC (big Endian-numbers the bytes from left to right)

c) The message after being inverted is not a general solution.d) The little numbers in boxes indicate the address of each byte

(need some type of deli-meter in order to understand the various representations involved)

e) integers and strings have to be handled differently!

21

Passing Reference Parameters• How are pointers being passed??

– Generally difficult problem…– One solution is forbid passing of pointers all together

(rather restrictive)– Some things can be done with arrays:

• Copy the array into the message and send it over..

• Call-by-reference is replaced by copy/restore..

• If the stubs know whether the buffer is an Input parameter to the server or an output parameter to the client optimizations can be done.

22

Parameter Specification and Stub Generation

En example.. • A procedure• The corresponding message.

•In order to hide the details of RPC some conventions (protocol) between theClient and the Servers have to be established.. •Representations (formats) of int, float, double, char, etc have to be •Be agreed upon.

RPC protocols can be established with IDL specs (that are compiledInto client/server stubs…)

23

Extensions of RPCs:Doors[Hamilton & Kougiouris 94]

•What if caller/callee are in the same machine?? (classic IPC problem)•Door is a generic name for a procedure in the address space of a server process & can be called by processes colocated with the server.•Requires support from the local OS.

Plus: allows the use of procedure calls mechanism for a distributed system.

Minus: developers still need to be aware whether a call is done locally or remotely

24

Asynchronous RPC

a) The interconnection between client and server in a traditional RPCb) The interaction using asynchronous RPC

2-12

25

Asynchronous RPC

• A client and server interacting through two asynchronous RPCs• One-way RPCs

2-13

26

Distributed Computing Environment(DCE)

Idea: take a group of machines (running Unix, OS2, Windows etc) add a layer of software and then be able to run distributed applications without disturbing the running of existing applications!

Some basic services offered such as:1. Distributed File Service (the worldwide file system)

2. Directory Service

3. Security Service

4. Distributed Time Service (why?)

27

Writing a Client and a Server

The steps in writing a client and a server in DCE RPC.

2-14

28

Binding a Client to a Server

• A client can call a server as long as the server is registered and ready to accept calls!

• Registration makes possible that the client can locate the server and bind to it.

Server location is down into two steps:• Locate the server’s machine• Locate the server (ie, the correct process on that machine!).

29

Binding of Clients and Servers• In order to achieve the step (2) the port for the

service needs to be known.. (or endpoint)

• Ports are used by servers as different entry points to different procedure calls.

• In DCE there is a daemon (DCE Daemon) that makes this look up between (server, endpoint)

• The server also registers with a directory machine(name of machine and IP number)

30

Binding a Client to a Server

Client-to-server binding in DCE.

2-15

•Example: a client wants to bind to a video server that is locally known as /local/multimedia/video/movies•DCE semantics

•Default: at-most-once operation•Otherwise: idempotent (done with the help of IDL)

•a remote Procedure call can be called a number of times.

31

Remote Object Invocation

• Object-Oriented Technology (CORBA, DCOM) is another way to develop distributed applications.

• RPCs could be applied to those frameworks as well.• Object::

– Encapsulates data– Methods (operations on data)– Methods are available via interfaces– The differentiation between object & interfaces is critical

as far as Distributed Systems concerns.– Interface can be on one machine – Object on another.

32

Distributed Objects

• Common organization of a remote object with client-side proxy.• Proxy = client stub ; Skeleton = Server Stub• The state of such distributed objects is NOT distributed!• A real-distributed object should be one that is physically distributed

across multiple machines.

2-16

33

34

35

Implicit/Explicit Binding of a Client to an Object

a) An example with implicit binding using only global references

b) An example with explicit binding using global and local references (the client first calls a a special function that binds the object before the client can invoke the object’s methods)

Distr_object* obj_ref; //Declare a systemwide object referenceobj_ref = …; // Initialize the reference to a distributed objectobj_ref-> do_something(); // Implicitly bind and invoke a method

(a)

Distr_object objPref; //Declare a systemwide object referenceLocal_object* obj_ptr; //Declare a pointer to local objectsobj_ref = …; //Initialize the reference to a distributed objectobj_ptr = bind(obj_ref); //Explicitly bind and obtain a pointer to the local proxyobj_ptr -> do_something(); //Invoke a method on the local proxy

(b)

36

Types of Remote Method Invocation (RMI)

• Static Invocation– Interfaces of an object are know when client application

is being developed.

• Dynamic Invocation– Be able to compose a methods invocation at run time!– invoke(object, method, in_params, out_params)– Example: append an integer int to a file object fobject for

which the object provides the method append• Static: fobejct.append(int)• Dynamic: invoke(fobject, id(append), int) where id returns an

identifier for the method append.• Dynamic Invocation: when is it useful?

37

Parameter Passing in RMI

Passing an object by reference or by value.

There are subtleties with RMI invocation (more so than with RPCs)References to remote/local objects are handled differently•When invoking a method with an object reference as parameter

•Reference is copied and passed as a value parameter only when it refers to a remote object•When the reference is made on a local (client) object, the object is copies as a whole and passed along with the invocation

38

Message Oriented Communication•RPCs & RMIs hide communication element of the interaction•In general both RPCs & RMIs are blocking•An alternative to those is communication based on messages•General organization for message-oriented communication is shown below

39

Example: Mail-Server

• Every host is connected to one mail-server (can think of it as being the “comm” server of the picture in the previous slide).

• A client interface allows users to get access to the messages (located on the mail servers).

• When a user submits an message, the host forwards the message its corresponding mail server.

• Mail servers forward/delete messages.

Such a system is an example of persistent communication

40

Mail-servers follow the Pony Express Model

Persistence has to do with the fact that the mail is stored as long as it takes to deliver it..

Transient Communication: if a message cannot be delivered to the next server/destination is discarded (typical of the transport layer – corresponds to store-and-forward router).

41

Possible Combinations(sync/async-persi/transi)

a) Persistent asynchronous communicationb) Persistent synchronous communication(msg actually delivered to the

receiving site)

2-22.1

42

Combos

c) Transient asynchronous communication (examples are UDP, asynchronous RPCs)

d) Receipt-based transient synchronous communication

2-22.2

43

Combos

e) Delivery-based transient synchronous communication at message delivery

f) Response-based transient synchronous communication (RPCs & RMIs mostly adhere to this type of communication).

44

Berkeley Sockets

Socket primitives for TCP/IP.

Primitive Meaning

Socket Create a new communication endpoint

Bind Attach a local address to a socket

ListenAnnounce willingness to accept connections

AcceptBlock caller until a connection request arrives

Connect Actively attempt to establish a connection

Send Send some data over the connection

Receive Receive some data over the connection

Close Release the connection

A socket is a communication endpoint to which the application from which can read data from or write data to.

45

How Berkeley SocketsWork

Connection-oriented communication pattern using sockets.

46

The MPI Interface

Some of the most intuitive message-passing primitives of MPI.

Primitive MeaningMPI_bsend Append outgoing message to a local send buffer

MPI_sendSend a message and wait until copied to local or remote buffer

MPI_ssend Send a message and wait until receipt starts

MPI_sendrecv Send a message and wait for reply

MPI_isend Pass reference to outgoing message, and continue

MPI_issendPass reference to outgoing message, and wait until receipt starts

MPI_recv Receive a message; block if there are none

MPI_irecv Check if there is an incoming message, but do not block

•MPI designed for parallel application programs (transient communication)•Communication happens among a group of known processes.

47

Message-Queuing Systems

Four combinations for loosely-coupled communications using queues.

•Offer intermediate storage capacity for msgs•Good for transferring bulky message (in seconds) than simple messages (usually in milliseconds).•Important: sender is provided with the guarantee that her message will ultimately be inserted in the recipient’s queue.•Servers/receivers can execute completely independent of each other

48

Message-Queuing Model: calls

Basic interface to a queue in a message-queuing system.

Primitive

Meaning

Put Append a message to a specified queue

GetBlock until the specified queue is nonempty, and remove the first message

PollCheck a specified queue for messages, and remove the first. Never block.

NotifyInstall a handler to be called when a message is put into the specified queue.

49

General Architecture of a Message-Queuing System

The relationship between queue-level addressing and network-level addressing.

• source queues, destination queues• database of queue names• queuing layer

50

General Architecture of a Message-Queuing System

The general organization of a message-queuing system with routers.

2-29

•Relays can be good (in a messaging system)•Usually the queue-to-location mapping is managed statically•Otherwise, routers have to be used (in this case, relays can be used to build scalable messaging systems).

51

Message Brokers

The general organization of a message broker in a message-queuing system.

•Brokers are used to convert to various message types.•For example convert message from X.400 to Internet messages•The database can handle many such conversion rules.

52

Stream-Oriented Communication

• Thus far, communication as long as it happens correctly is all we mind. Timing of this communication was not the issue.

• Timing is important in certain types of communication …– CD quality audio transmission ( 16bit samples at

44,1kHz)– Video streams

53

Streams - Definitions

• Information Representation– Text with ASCII/Unicode

– Images with GIF/JPEG

– Audio Streams with 16bit samples using PCM

• Continuous (Representation) Media: temporal relationship among the different data items are fundamental to interpret what the data means.– Motion requires 30-40 msec per image (if represented as a

sequence of images).

• Discrete(Representation) Media: temporal relations among data items are not of essence – Representations of text, still-images, code, files, etc.

54

Data Streams• Data stream: a sequence of data units

• Timing is essential into continuous data streams.

• Asynchronous Transmission Mode– Data are xmited one after the other with no “timing” strings

attached.

• Synchronous Transmission Mode– There is a maximum end-to-end delay defined for every unit in the

data stream.

• Isochronous Transmission Mode– Data have to be transferred on time

– There are minimum and maximum end-to-end delay (jitter)

55

Data Streams

Setting up a stream between two processes across a network.

•Simple stream: single sequence of data•Complex stream: consists of several related simple streams (known as substreams)•A stream is often considered a (virtual) connection between a source and a sink.

56

Data Streams with Direct Connection between Source and Sink

Setting up a stream directly between two devices.

57

What if the sink is a multiparty?

• Receivers may have different requirements

• Filters have to be attached…

The data stream is multicast to several (receivers) sinks.

58

Specifying QoS

A flow specification.

Characteristics of the Input Service Required

•maximum data unit size (bytes)•Token bucket rate (bytes/sec)•Toke bucket size (bytes)•Maximum transmission rate (bytes/sec)

•Loss sensitivity (bytes)•Loss interval (sec)•Burst loss sensitivity (data units)•Minimum delay noticed (sec)•Maximum delay variation (sec)•Quality of guarantee

Time-dependent requirements are collectively known as Quality of Service (QoS) requirements.

How one could define such a QoS? .. Via a flow spec

59

Partridge’s Model expressed with a Token Bucket Algorithm

The principle of a token bucket algorithm.

•This model specifies how the stream will shape its network traffic•Idea: tokens appear in the network at a constant rate.

•When the bucket fills, tokens are dropped.•Each time the application wants to pass N bytes removes N/k tokens from the bucket (each token represents k bytes).

60

Setting Up a Stream- RSVP Protocol

The basic organization of RSVP for resource reservation in a distributed system.

61

Synchronization Mechanisms

The principle of explicit synchronization on the level data units.

•Complex Streams have to be synchronized.. •How is this done?

62

Synchronization Mechanisms

The principle of synchronization as supported by high-level interfaces.

Multimedia Middleware offers a collection of interfaces• interfaces control audio/video streams• control devices such as mics, cameras, monitors• each device/stream has its own interface (used to write synchronization handlers).