A Scalable Content-Addressable Network Authors: S. Ratnasamy, P. Francis, M. Handley, R. Karp, S. Shenker University of California, Berkeley Presenter:

A Scalable Content-Addressable Network

Authors: S. Ratnasamy, P. Francis, M. Handley, R. Karp, S. Shenker

University of California, Berkeley

Presenter: Andrzej Kochut

Main features of CAN

Distributed, Internet-scale hash tableNo form of centralized controlScalable – amount of information stored at

each node independent of the total number of nodes in the system

Does not impose any form of hierarchical name structure

Can be implemented at the application level

Basic operations of CAN

• Inserting, updating, deleting of (key,value) pairs

• Retrieving value associated with a given key

• Adding new nodes to CAN

• Handling departing nodes

• Dealing with node failures

Basic CAN design

Design of CAN• Virtual d-dimensional coordinate space on a

d-torus• At any instant the entire coordinate space is

partitioned among all the nodes in the system

• Each node contains chunk of the hash table and information about its neighbors in the d-space

• Uniform hash function is used to map key values to points in the d-dimensional space

Example 2-space partition

Design of CAN

• To store (K,V) pair the hash value P of K is computed and the pair is stored in the node that owns point P

• To retrieve the value given K, the hash value P of K is computed. If the requesting node does not own point P, the request is routed towards the node that owns P

• The set of immediate neighbors serves as a coordinate routing table that enables routing between arbitrary points in the d-space

Routing in CAN• Intuitively – following the straight path through

the Cartesian space from source to destination• Node maintains coordinate routing table that holds

IP addresses and zones’ coordinate of its neighbors in the space

• Two nodes are neighbors if their coordinate spans overlap along d-1 dimensions and abut along one dimension

• CAN message contains destination coordinate. Node greedy forwards it to the neighbor with coordinates closest to the destination coordinate

Routing in CAN

• For the d-dimensional space equally partitioned into n nodes the average routing path is (d/4)*n(1/d)

• Individual nodes maintain 2d neighbors

• The path length growth proportionally to the O(n(1/d))

• Many different routes between two points

CAN construction

• New node is allocated its own portion of the coordinate space in three steps:– Find a node already in the CAN – look up the

CAN domain name in DNS– Pick zone to join to and route request to its

owner using CAN mechanisms – Split the zone between old and new node– The neighbors of the split zone must be notified

so the routing can include the new node

Finding and splitting a zone

• Randomly choose a point P in the space• Send join request destined for P• Route the request using CAN mechanisms• Split the zone occupied by the owner of P

assuming certain ordering of the dimensions, i.e. first X then Y

• Transfer (key, value) pairs from the half of the zone to the new node

Joining the routing

• Update neighbor information in the old and new node

• Inform all neighbors about changes in the zone – every node in the system sends immediate update message followed by periodic refreshes

Node departure and recovery• Normal procedure – explicit hand over of

(key,value) database to one of the neighbors• Node failure – immediate takeover procedure:

– Failure detected as a lack of update messages– Each neighbor starts timer with proportion to the

node’s zone size– After timer expires the node extends its own zone to

contain the failed neighbor’s zone and sends TAKEOVER message to all failed node’s neighbors

– On receive of the TAKEOVER node cancels its timer if the sender’s zone size is smaller than his own. Otherwise it sends it’s own TAKEOVER message.

Recovery cont.

• CAN state may become inconsistent if multiple adjacent nodes fail simultaneously

• In such cases perform an expanding ring search for any nodes residing beyond the failure region (?)

• If it fails initiate repair mechanism (?)• If a node holds more than one zone initiate the

background zone-reassignment algorithm

Drawbacks of the basic CAN design

• Not fault tolerant

• No recovery mechanism

• Lack of load-balancing mechanisms (no caching and replication)

• Does not consider the underlying IP topology while building overly network

Design improvements

Design improvements

• Realities: multiple coordinate spaces

• Topologically-sensitive construction of the CAN

• Multiple hash functions

• Better routing metrics

• Overloading coordinate zones

• More uniform partitioning

• Caching and replication

Multiple coordinate spaces

• Each node is assigned a different zone in each reality (coordinate space)

Improved robustness - point P is unreachable only if P in all realities is unreachable

Improved routing – to forward a message, a node checks all its neighbors on each reality and forwards the message to the neighbor with coordinates closest to the destination

Increased per-node state

Topologically-sensitive construction of the CAN

• Assumes existence of a well known set of machines (i.e. DNS root name servers) called landmarks

• Each node that joins the CAN adds itself to the region associated with its perceived ordering of landmarks

Reduces latency stretch (ratio of the CAN latency to the IP latency)

Leads to unbalanced partition of space between nodes

Multiple hash functions

• Many different hash functions to map (key, value) pairs to the points in space

• Data replicated accordinglyImproved fault toleranceIncrease in the size of (key,value) database

and in the size of the query trafficData consistency issues

Effects of the improvements

Effects of the improvements (cont.)

Better routing metrics

• Each node measures the network-level RTT to each of its neighbors

• Message forwarded to the neighbor with the highest progress to RTT ratio

Overloading coordinate zones

• More than one node associated with each zone• Each node maintains list of its peers in the zone

and neighboring information about selected node in each of its neighboring zones

• Adding new nodes – if the zone to which a new node joins contains less than MAXPEERS nodes, the new node joins the zone without any space-splitting

• Neighboring nodes in the adjacent zones are chosen based on the RTT

Advantages of the zone overloading

• Reduced path length

• Reduced per-hop latency – node has multiple choices in neighbors selection

• Improved fault-tolerance

More uniform partitioning• Attempt to have equal

partitioning of the space between nodes

• Existing occupant of the zone that a new node joins redirects the request to the zone with the highest volume among his and all of his neighbors

Not true load balancing – does not reflect popularity of the data

Caching and replication

• Each node maintains cache of the most popular requests

• A node that is being overloaded by requests to the particular data key replicates this key to all of its neighbors

No description of the way to check validity of the cached and replicated data

Performance tests

• System size – 218 nodes

• Topology generated by GT modeler

Results only from simulations

Performance test results

Conclusion and presenter’s opinion

• Addresses the problems of scalable routing and indexing

• Test results only from simulationUnresolved security issues (i.e. denial of service

attacks)Lack of search techniques (i.e. keyword searching)Not clearly specified recovery techniquesNot clearly specified data consistency issues

(replication, caching)Does not address the problem of node’s connection

quality