Apache ZooKeeper TechTuesday

Apache ZooKeeper

Andrei Savu @TechTuesday

Why use it? What to expect in the future?

Outline

Why use it?Crash CoursePractical Example

What to expect in the future (3.4.0 release)?GSoC 2010New ContribWork in Progress

Crash Course

What is ZooKeeper?

A highly available, scalable, distributed, configuration, consensus, group membership,

leader election, naming and coordination service.

What is ZooKeeper? (2)

replicated in memory tree data structuresomehow similar to a file systemno partial read / writesno renamesordered updatesstrong persistence guaranteesconditional updates (version)watches for data changesephemeral nodesgenerated file names

ZooKeeper Data Model

hierarchical namespaceeach znode has data and childrendata is read and written in its entirety

Basic ZooKeeper API

string create(path, data, acl, flags)

delete(path, expected_version)

stat set_data(path, data, expected_version)

(data, stat) get_data(path, watch)

stat exists(path, watch)

string[] get_children(path, watch)

ZooKeeper Service

Facts: 1) all servers store a copy of the data in memory 2) the leader is elected at startup 3) followers respond to clients 4) all updates go through the leader 5) responses are sent when a majority of servers have persisted the change

Practical Example

Distributed Queue (Python)

http://www.cloudera.com/blog/2009/05/building-a-distributed-concurrent-queue-with-apache-zookeeper/ http://github.com/henryr/pyzk-recipesRetry operation on ConnectionLoss:

http://github.com/andreisavu/pyzk-recipes

GSoC 2010

3 projects / 5 months

1. Monitoring & Web-based interface

Status: Committed to the trunk

1. JIRA: ZOOKEEPER-7012. Progress Tracking Wiki3. Monitoring for Ganglia, Nagios and Cacti

1. contrib / monitoring2. 'mntr' 4letter word

4. Web interface available as a Hue application1. contrib / huebrowser2. complete install instructions3. requirements: rest gateway, Hue 1.0

2. Read-Only Mode

Status: Under Review (Ready to be committed)

1. JIRA: ZOOKEEPER-7042. Progress Tracking Wiki3. Description: "When a ZooKeeper server loses contact with

over half of the other servers in an ensemble ('loses a quorum'), it stops responding to client requests. For some applications, it would be beneficial if a server still responded to read requests when the quorum is lost, but caused an error condition when a write request was attempted."

3. Failure Detector Model

Status: Under Review

1. JIRA: ZOOKEEPER-7022. Progress Tracking Wiki3. Detectors: Phi Accrual, Chen, Bertier, Fixed Heartbeat4. Why? Check the concluding remarks on the wiki. 5. Conclusion snippet: "in scenarios where we have a

changing network behavior, such in a WAN, the adaptive methods can be a good pick"

Contrib

Large Scale Pub/Sub (hedwig)

1. JIRA: ZOOKEEPER-7752. Uses ZooKeeper and BookKeeper3. Committed to the trunk4. Developed at Yahoo! Research5. Used for PNUTS cross data center replication6. http://vimeo.com/13282102

Work in Progress

only some interesting JIRAs

#834 Children for ephemerals

JIRA: ZOOKEEPER-834Allow ephemeral nodes to have children owned by the same session. Useful when publishing status information. No need to do serialization for basic data structures (hash tables)Similar to /proc in *nix systems.Examples: /agent-01/ip, /agent-01/memory, /agent-01/load

#829 /zookeeper/sessions/* 

JIRA: ZOOKEEPER-829Requested by HBase developers: " we'd like the ability to forcible expire someone else's ZK session "

Plenty of bug fixes

Join the community!

Resources

http://hadoop.apache.org/zookeeper/http://wiki.apache.org/hadoop/ZooKeeper/ProjectDescriptionhttp://wiki.apache.org/hadoop/ZooKeeper/Tao

Thanks! Questions?