Graphs, Edges & Nodes: Untangling the Social Web

Graphs, Edges & Nodes

Untangling the social web.

Wednesday, March 9, 2011

What’s a graph?

Simple

At most one edge between any pair of nodes.

Multigraph

Multiple edges between vertices allowed.

Pseudograph

Self-loops are permitted.

G = (V, E)

What’s a node?

vertexpoint

junction0-simplex

What’s an edge?

arcbranch

linelink

1-simplex

Directed

Undirected

Data Structures

(Finite simple graph)

Adjacency Matrix(2d array)

0 1 1 1

1 0 0 0

1 0 0 1

1 0 1 0

vertices

Adjacency Matrix(2d array)

0 1 1 1

1 0 0 0

1 0 0 1

1 0 1 0

vertices

(Finite simple graph)

[1, 2, 3, 4]234

Array entries (vertices) point to singly linked-lists

Visualizations

You are here.

(Graph does not include Justin Bieber)

Social Graphs

User-based item recommendations

People

Recommend items to me that are popular amongst my friends

People

(friends)

People

(friends)

People

(friends)

People

(friends)(me)

People

(friends)(me)

People

(friends)(me)

2-step path on homogeneous bipartitegraph.

Strong Connection Problem (SCP)

There are many of these ‘fundamental’ graph units:

- tripartite graphs (user/asset/tag)- folksonomies- multicolor-multiparity graph- etc.

Graph Storage Engines

Neo4j“An embedded, disk-based, fully transactional Java persistence engine that

stores data structured in graphs rather than in tables.”

http://neo4j.org

HypergraphDB“A general purpose, extensible, portable, distributed, embeddable, open-source

data storage mechanism. It is a graph database designed specifically for artificial intelligence and semantic web projects.”

http://kobrix.org/hgdb.jsp

Special Purpose Storage Engines

FlockDB“FlockDB is a database that stores graph data, but it isn't a database

optimized for graph-traversal operations. Instead, it's optimized for very large adjacency lists, fast reads and writes, and page-able set arithmetic

queries.”

http://engineering.twitter.com/2010/05/introducing-flockdb.html

Redis“Redis is an advanced key-value store. [...] the dataset is not volatile, and values can be strings, exactly like in memcached, but also lists, sets, and ordered sets. All this data types can be manipulated with atomic operations to push/pop elements, add/remove elements, perform server side union, intersection, difference between sets, etc.”

http://code.google.com/p/redis

A Redis Friends/Followers Example

Redis makes you think in terms of datastructures,and operations on those structures.

Set:Finite (for our cases) collection of objects in which order has no significance and multiplicity is generally ignored.

S = { Alice, Bob, Carol }

List:Finite (for our cases) collection of objects in which order *is* significant and multiplicity is allowed.

L = [ X, Y, X, Z, Q]

SET uid:1000:username jperras

Insert a user into a set

Command Key Value

Use sets for denoting my followers/peopleI follow.

SADD uid:1000:following 1001SADD uid:1001:followers 1000

Adding a new follower

Command Key Value

Posting Updates

$r = Redis();$postid = $r->incr("global:nextPostId");$post = $User['id'] ."|". time() ."|". $status;$r->set("post:$postid", $post);$followers = $r->smembers("uid:".$User['id'].":followers");

if ($followers === false) $followers = Array();$followers[] = $User['id']; /* Add the post to our own posts too */

foreach($followers as $fid) { $r->push("uid:$fid:posts", $postid, false);}# Push the post on the timeline, and trim the timeline to the# newest 1000 elements.$r->push("global:timeline", $postid, false);$r->ltrim("global:timeline",0,1000);

Common followers? - Set intersections!

SINTER users:1000:followers users:1000:followers

Command Key 1 Key 2

A MySQL Example

(simplified)

# Mutual Friendsselect f.friend_id from friends f join friends m on m.user_id = f.friend_id and m.friend_id = f.user_idwhere f.user_id = 1234

# Following (for directed graphs)select f.friend_id from friends f left join friends m on m.user_id = f.friend_id and m.friend_id = f.user_id where f.user_id = 1234 and m.user_id is null;

# Followers (for directed graphs)select m.friend_id from friends f left join friends m on m.user_id = f.friend_id and m.friend_id = f.user_id where f.friend_id = 1234 and m.user_id is null

Not too bad.

# Mutual Friendsselect f.friend_id from friends f join friends m on m.user_id = f.friend_id and m.friend_id = f.user_idwhere f.user_id = 1234

# Following (for directed graphs)select f.friend_id from friends f left join friends m on m.user_id = f.friend_id and m.friend_id = f.user_id where f.user_id = 1234 and m.user_id is null;

# Followers (for directed graphs)select m.friend_id from friends f left join friends m on m.user_id = f.friend_id and m.friend_id = f.user_id where f.friend_id = 1234 and m.user_id is null

Relational databases can work for the simplestof cases, but are not always the best solution for

many graph operations/algorithms.

Graphs and graph-databases are onlygoing to be more and more useful.

However, graph algorithms are hard.

So don’t write your own.

And make sure you use a persistent storage enginethat is best suited for the type of queries

you will be performing.

Resources

The Algorithm Design Manual, Steve S. Skiena

Programming Collective Intelligence, Toby Segaran

Introduction to Algorithms, Cormen, Leiserson, Rivest

Resources

@jperras

Graph of the internet, circa 2003: http://www.duniacyber.com/freebies/education/what-is-internet-lookslike/ (built from partial troll of public servers using traceroute)

My real friends for letting me use their Facebook profile images.

Photo Credits

Large Scale Graph Algorithms (class lectures), Yuri Lifshits, Steklov Institute of Mathematics at St. Petersburg

http://mathworld.wolfram.com/Set.html

Programming Collective Intelligence, Toby Segaran

The Algorithm Design Manual, Steve S. Skiena

References

Graphs, Edges & Nodes: Untangling the Social Web

peoplefriends items

graph database

graph storageengineswednesday

graphtraversal operations

redis redis

finite simple graphwednesday

multicolormultiparity

homogeneous bipartite

Technology

Drawing graphs with · Drawing graphs with dot ... nates of...

Untangling CCC bureaucracy

Open Access Research Sensitivity analysis for mistakenly...

Interaction Content Aware Network Embedding via Co-embedding...

Activity Diagrams. Recap Activity Diagrams – When to use?....

Advanced Microeconomics · Advanced Microeconomics...

Untangling Regulations

Message Passing Systems A Formal Model. The System Topology....

Groove - quick reference · 2012. 11. 13. · Graph: A set....

Untangling Names

Treesrupesh/teaching/pds/jul19/5-trees.pdf · Trees with...

Graphs and Networks in Linear Algebra · For computer...

Graph Algorithms: Classification William Cohen. Outline Last...

Neural Networks - Carnegie Mellon School of … · Neural.....

Network Flow - laber/07maxflow.pdf · 4 Flow network....

Probabilistic Models of Structured Data -...