Top Banner
Pastry Pastry : : Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware 2001
32

Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

Dec 13, 2015

Download

Documents

Violet Marshall
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

PastryPastry: : Scalable, decentralized object location androuting for large-scale peer-to-peer systems

Antony Rowstron and Peter Druschel, Middleware 2001

Page 2: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

2

OutlineOutline

IntroductionDesign of Pastry

– Node state & routing– Pastry API– Self-organization and adaptation– Locality

Experimental ResultsDiscussion

Page 3: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

3

Introduction:Introduction: What is Pastry? What is Pastry?

It’s a scalable, distributed, decentralized object location and routing substrate

Serves as a general substrate for building P2P applications: SCRIBE, PAST,…etc.

Seeks to minimize distance messages travelPastry’s main capability

Page 4: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

4

Pastry NodePastry Node Represented by 128-bit randomly chosen nodeId (Hash of IP

or public key) NodeId is in base 2b (b is a configuration parameter; b typical

value 2 or 4) Evenly distributed nodeIds along the circular namespace (0-

2128 – 1 space). Routes a message in O(log N) steps to destination

– N: size of network

Node state contains:– Leaf Set ( L )– Routing table ( R )– Neighborhood Set ( M )

Page 5: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

5

Design of Pastry:Design of Pastry: Node stateNode state

Leaf set: L/2 Numerically closest nodes (L is a configuration parameter = 16, 32 typically )

Routing Table (Prefix-based)

Neighborhood Set: M physically closest nodes

Page 6: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

6

Pastry node state (Leaf Set)Pastry node state (Leaf Set)

Serves as a fall back for routing table and contains:– L/2 numerically closest and larger nodeIds– L/2 numerically closest and smaller nodIds

Size of L is typically 2b or 2 x 2b

Nodes in L are numerically close (could be geographically diverse)

Page 7: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

7

Pastry node statePastry node state: : Neighborhood Neighborhood set (M)set (M)

Contains the IP addresses and nodeIds of closest nodes according to proximity metric

Size of |M| is typically 2b or 2x2b

Not used in routing, but instead for maintaining locality properties

Page 8: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

8

Node state:Node state: Routing TableRouting Table Matrix of Log2b N rows and 2b – 1 columns (N is the

number of nodes in the network)

Entries in row n match the first n digits of current nodeId AND

Column number follows matched digits: Format: matched digits–column number–rest of ID

Log2b N populated on average

Page 9: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

9

Node10233102 Node10233102 (2),(2), ( (b = 2, l = 8)b = 2, l = 8)

0 1 2 302212102 22301203 31203203

11301233 12230203 1302102210031203 10132102 1032330210200230 10211302 102230210230322 10231000 1023212110233001 10233232

10233120

Page 10: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

10

RoutingRouting(2)(2)::

If message with key D is within range of leaf set, forward to numerically closest leaf

Else forward to node that shares at least one more digit with D in its prefix than current nodeId

If no such node exists, forward to node that shares at least as many digits with D as current nodeId but numerically nearer than current nodeId

Page 11: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

11

Routing Messages Routing Messages

D: Message KeyLi: ith closest NodeId in leaf setshl(A, B): Length of prefix shared by nodes A and BRi

j: (j, i)th entry of routing table

(1) Node is in the leaf set

(2) Forward message to a closer node (Better match)

(3) Forward towards numericallyCloser node (not a better match)

Source: Rowstron & Drushel, 2001

Page 12: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

12

Routing Example: Routing Example:

Source: www.scs.cs.nyu.edu/V22.0480-005/notes/l24.pdf

Page 13: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

13

Routing Performance:Routing Performance: (1) If key is within leaf set:

– target one hop away

(2) If key has to be routed:– Number of nodes with longer prefix decreases by 2b

(3) Key is not covered by the leaf set (i.e., failed)– With high probability, one more hop needed

Thus: Number of routing steps needed log 2b N

Page 14: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

14

Pastry APIPastry API::

operations exported by pastry:

nodeId = pastryInit(Credentials, Application )– Causes a Pastry node to join the network with state

initialization. Other application specific information is also established.

route(msg, key)– Routes the message to another node which is

numerically closest to the key

Page 15: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

15

Pastry APIPastry API

Operations exported by the application working on top of Pastry

deliver(msg,key)– Called when local node is numerically closest to the key

forward(msg, key, nextId)– forward a message from the local nodeId to the next

nodeId ( nextId = Null if local node is final) newLeafs( leafSet):

– Updates the leaf set

Page 16: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

16

Pastry node joinPastry node join

X = new node, Z = numerically closest node, A = bootstrap (A is close in proximity space to X)

X sends a join message to A with target nodeId X A forwards to B C… Stops at Z, numerically closest to X’s nodeId

In process, A,B,…,Z send their state tables to X

Page 17: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

17

Node JoinNode Join

X’s neighborhood set (NS) = A’s NS X’s Leaf Set = Z’s leaf set X’s routing table is filled as follows:

– X’s Row 0 = A’s row 0 (X0 = A0)

– X’s Row 1 = B’s row 1 (X1 = B1)

– …etc.

X sends its state to every node in its state tables ( Leaf set, neighborhood set, and routing table)

Page 18: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

18

Node JoinNode Join: Example : Example

www.scs.cs.nyu.edu/V22.0480-005/notes/l24.pdf

Source: www.scs.cs.nyu.edu/V22.0480-005/notes/l24.pdf

Page 19: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

19

Node departure Node departure (2)(2)

Invalid nodes in leaf set: detected by heartbeat monitor– Repair by inserting node from another leaf’s LS

Heartbeat for neighborhood set (NS)– Query all NS members for their NS tables, choose

replacement according to proximity metric Invalid routing entries detected when attempting

to route– Query nodes in row for replacement entry, if failed– Query successive rows until success

Page 20: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

20

Node failure in routing table: exampleNode failure in routing table: example

If node in red failsIf node in red fails

Source: www.cs.cornell.edu/courses/ cs514/2003fa/CS514-fa03-lec26v0.pdf

Page 21: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

21

Locality in PastryLocality in Pastry Based on proximity metric (i.e., No. of IP hops,

geographic distance) Proximity space is assumed to be Euclidean The route chosen for a message is likely to be

“good“ with respect to the proximity metric We will discuss locality regarding:

– Routing table locality

– Route locality

– Locating the nearest among k nodes

Page 22: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

22

Locality in Routing tablesLocality in Routing tables

Invariant: “all routing table entries refer to a node that is near the present node, according to the proximity metric, among all live nodes with a prefix appropriate for the entry.”

We wish to maintain the invariant when adding new nodes.

X joins; A is close to X; X0 = A0, so locality holds in X’s routing table

X1 = B1. Entires in B1 (row 1 of X) are close to B, but are they necessarily close to X?

Page 23: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

23

Locality in routing tableLocality in routing table

Entries of B1 are reasonable close to X Why?

– A is much closer to B than entry in B1 to B because

every time we choose from an exponentially decreasing set of nodes

To improve proximity approximation:– X Queries nodes in routing table and neighborhood set

for their state

– Compares distances (from routing table entries) and update route entries with closer nodes if found.

Page 24: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

24

Route localityRoute locality At each routing step the message is moved closer to

the destination in the: – nodeId space (numerically closer nodes)– proximity space: message travels the least possible distance

Given that:– A message routed from A to B at a distance d cannot be

routed to a node with a distance of less than d from A. (follows from routing procedure)

– Expected distance traveled increases exponentially Though shortest path is not guaranteed, we still get a

good route.

Page 25: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

25

Locality among Locality among kk nodes nodes In some Pastry-based applications, object is replicated

on k nodes on its route (during insertion) In prefix-base routing: goal is to reach any of k

numerically closest nodes that has a copy of object May miss nearby nodes with different prefix Use heuristic to determine when close to k nearest

nodes– Based on density of nodeIds that store object; using local info

– Switch to numerically closest address

Page 26: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

26

Arbitrary node failureArbitrary node failure

Node continues to be responsive, but behaves incorrectly or maliciously.

Repeated queries fail each time because they normally take the same route.

How to solve it? Use randomized routing– The choice among multiple nodes that satisfy

the routing criteria should be made randomly

Page 27: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

27

Routing PerformanceRouting Performance

|L|=16 * b=4 * |M|=32 * 200,000 lookups

Source: Rowstron & Drushel, 2001

Page 28: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

28

Pastry routingPastry routing

Source: Rowstron & Drushel, 2001

Page 29: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

29

Routing with failuresRouting with failures

Source: Rowstron & Drushel, 2001

Page 30: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

30

Pastry locality Pastry locality

|L|=16 * b=4 * |M|=32 * 200,000 lookups Source: Rowstron & Drushel, 2001

Page 31: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

31

SummarySummary

Pastry is a generic P2P object location and routing substrate

Distributed, and scales wellUsed in developing applications like file

storage, global file sharing,...etc.Considers locality when routing messeges

Page 32: Pastry: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware.

32

ReferencesReferences

(1) A. Rowstron and P. Druschel, "Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems".  IFIP/ACM International Conference on Distributed Systems Platforms (Middleware), Heidelberg, Germany, pages 329-350, November, 2001

(2) Jeff Odom slides: http://x1.cs.umd.edu/818/docs/pastry.ppt