NOSQLEU - Graph Databases and Neo4j

Graph Databasesand Neo4j

Tobias IvarssonHacker @ Neo Technology

twitter: @thobe / #neo4jemail: [email protected]: http://www.neo4j.org/web: http://www.thobe.org/

http://twitter.com/thobe

http://twitter.com/thobe

http://search.twitter.com/search?q=neo4j

http://search.twitter.com/search?q=neo4j

mailto:[email protected]?subject=

mailto:[email protected]?subject=

http://www.neo4j.org

http://www.neo4j.org

http://www.thobe.org

http://www.thobe.org

2

Four trends

NOSQL - Why now?

2006 2007 2008 2009 2010

0

250

500

750

1000

161253

397

623

988ExaBytes (10¹⁸) of data stored per year

3

Trend 1: Data size

Data source: IDC 2007

Each year more and more digital data is created. Over two years we create more digital data than all the data created in history before that.

http://www.internetnews.com/stats/article.php/3663641

http://www.internetnews.com/stats/article.php/3663641

Trend 2: Connectedness

4

Text documents

1990

Info

rmat

ion

conn

ectiv

ity Folksonomies

Tagging

User-generated content

Wikis

RSS

Blogs

Hypertext

2000 2010 2020

web 1.0 web 2.0 “web 3.0”

Ontologies

RDF

GiantGlobal

Graph (GGG)

Over time data has evolved to be more and more interlinked and connected.Hypertext has links,Blogs have pingback,Tagging groups all related data

Trend 3: Semi-structure

5

๏ Individualization of content

• In the salary lists of the 1970s, all elements had exactly one job

• In the salary lists of the 2000s, we need 5 job columns! Or 8? Or 15?

๏All encompassing “entire world views”

• Store more data about each entity

๏Trend accelerated by the decentralization of content generation that is the hallmark of the age of participation (“web 2.0”)

Trend 4: Architecture

6

DB

Application

1980s: Mainframe applications


7

DB

Application

1990s: Database as integration hub

Application Application

DBDB DB


8

Application

2000s: (moving towards) Decoupled serviceswith their own backend

Application Application

Why NOSQL Now?

๏Trend 1: Size

๏Trend 2: Connectedness

๏Trend 3: Semi-structure

๏Trend 4: Architecture

9

RDBMS performance

10Data complexity

Perfo

rman

ce

Majority ofWebapps

Social network

Semantic Trading

Salary List

}custom

Relational database

Requirement of application

We are building applications today that have complexity requirements that a Relational Database cannot handle with sufficient performance

Scaling to size vs. Scaling to complexity

11

Size

Complexity

Key/Value stores

Bigtable clones

Document databases

Graph databases

> 90% of use cases

Billions of nodesand relationships

12

Graph Databases focuses on structure of data

Graph databases focus on the structure of the data, scaling to the complexity of the data and of the application.

What is Neo4j?๏Neo4j is a Graph Database

•Non-relational (“#nosql”), transactional (ACID), embedded

•Data is stored as a Graph / Network

‣Nodes and relationships with properties

‣“Property Graph” or “edge-labeled multidigraph”

• Schema free, bottom-up data model design

๏Neo4j is Open Source / Free (as in speech) Software

•AGPLv3

•Commercial (“dual license”) license available

‣First server is free (as in beer), next is inexpensive 13

Prices are available at http://neotechnology.com/

Contact us if you have questions and/or special license needs (e.g. if you want an evaluation license)

http://search.twitter.com/search?q=nosql

http://search.twitter.com/search?q=nosql

http://wiki.github.com/tinkerpop/gremlin/defining-a-property-graph

http://wiki.github.com/tinkerpop/gremlin/defining-a-property-graph

http://en.wikipedia.org/wiki/Multigraph

http://en.wikipedia.org/wiki/Multigraph

http://neotechnology.com


More about Neo4j๏Neo4j is stable

• In 24/7 operation since 2003

๏Neo4j is in active development

•Neo Technology received VC funding October 2009

๏Neo4j delivers high performance graph operations

• traverses 1’000’000+ relationships / secondon commodity hardware

14

The Neo4j Graph data model

15

•Nodes•Relationships between Nodes•Relationships have Labels•Relationships are directed, but traversed at equal speed in both directions•The semantics of the direction is up to the application (LIVES WITH is reflexive, LOVES is not)•Nodes have key-value properties•Relationships have key-value properties


15



15

LIVES WITHLOVES

OWNSDRIVES



15

LIVES WITHLOVES

OWNSDRIVES

LOVES



15

LIVES WITHLOVES

OWNSDRIVES

LOVESname: “James”age: 32twitter: “@spam”

name: “Mary”age: 35

brand: “Volvo”model: “V70”



15

LIVES WITHLOVES

OWNSDRIVES

LOVESname: “James”age: 32twitter: “@spam”

name: “Mary”age: 35

brand: “Volvo”model: “V70”

item type: “car”


Graphs are all around us

A B C D ...

1

2

3

4

5

...

17 3.14 3 17.79333333333

42 10.11 14 30.33

316 6.66 1 2104.56

32 9.11 592 0.492432432432

2153.175765766

16

Even if this spreadsheet looks like it could be a fit for a RDBMS it isn’t:•RDBMSes have problems with extending indefinitely on both rows and columns•Formulas and data dependencies would quickly lead to heavy join operations


17

A B C D ...

1

2

3

4

5

...

17 3.14 3 = A1 * B1 / C1

42 10.11 14 = A2 * B2 / C2

316 6.66 1 = A3 * B3 / C3

32 9.11 592 = A4 * B4 / C4

= SUM(D2:D5)

With data dependencies the spread sheet turns out to be a graph.


17

A B C D ...

1

2

3

4

5

...

17 3.14 3 = A1 * B1 / C1

42 10.11 14 = A2 * B2 / C2

316 6.66 1 = A3 * B3 / C3

32 9.11 592 = A4 * B4 / C4

= SUM(D2:D5)

With data dependencies the spread sheet turns out to be a graph.


18

17 3.14 3 = A1 * B1 / C1

42 10.11 14 = A2 * B2 / C2

316 6.66 1 = A3 * B3 / C3

32 9.11 592 = A4 * B4 / C4

= SUM(D2:D5)

If we add external data sources the problem becomes even more interesting...


18

17 3.14 3 = A1 * B1 / C1

42 10.11 14 = A2 * B2 / C2

316 6.66 1 = A3 * B3 / C3

32 9.11 592 = A4 * B4 / C4

= SUM(D2:D5)

If we add external data sources the problem becomes even more interesting...

Graphs are whiteboard friendly

19Image credits: Tobias Ivarsson

An application domain model outlined on a whiteboard or piece of paper would be translated to an ER-diagram, then normalized to fit a Relational Database.With a Graph Database the model from the whiteboard is implemented directly.


19

1

*

1

*

*

1*

1

*

*

Image credits: Tobias Ivarsson



19

thobe

Wardrobe Strength

Joe project blog

Image credits: Tobias Ivarsson


Hello Joe

Neo4j performance analysis

Modularizing Jython

Query Languages๏Traversal APIs

•Neo4j core traversers

•Blueprint pipes

๏SPARQL - “SQL for linked data” - query by graph pattern matching

SELECT ?person WHERE {?person neo4j:KNOWS ?friend .?friend neo4j:KNOWS ?foe .?foe neo4j:name "Larry Ellison" .

}

๏Gremlin - “perl for graphs” - query by traversal

./outE[@label='KNOWS']/inV[@age > 30]/@name

20

Find all persons that KNOWS a friend that KNOWS someone named “Larry Ellison”.

Give me the names of all the people I know that are older than 30.

Data manipulation APIGraphDatabaseService graphDb = getGraphDbInstanceSomehow();

// Create Thomas 'Neo' AndersonNode mrAnderson = graphDb.createNode();mrAnderson.setProperty( "name", "Thomas Anderson" );mrAnderson.setProperty( "age", 29 );

// Create MorpheusNode morpheus = graphDb.createNode();morpheus.setProperty( "name", "Morpheus" );morpheus.setProperty( "rank", "Captain" );morpheus.setProperty( "occupation", "Total bad ass" );

// Create relationship representing they know each othermrAnderson.createRelationshipTo( morpheus, RelTypes.KNOWS );// ... similarly for Trinity, Cypher, Agent Smith, Architect

21

Data manipulation APIGraphDatabaseService graphDb = getGraphDbInstanceSomehow();

// Create Thomas 'Neo' AndersonNode mrAnderson = graphDb.createNode();mrAnderson.setProperty( "name", "Thomas Anderson" );mrAnderson.setProperty( "age", 29 );

// Create MorpheusNode morpheus = graphDb.createNode();morpheus.setProperty( "name", "Morpheus" );morpheus.setProperty( "rank", "Captain" );morpheus.setProperty( "occupation", "Total bad ass" );

// Create relationship representing they know each othermrAnderson.createRelationshipTo( morpheus, RelTypes.KNOWS );// ... similarly for Trinity, Cypher, Agent Smith, Architect

21

Transaction tx = graphDb.beginTx();try {

tx.success();} finally {

tx.finish();}

since: “meeting the oracle” since: “a year before the movie”cooperates on: “The Nebuchadnezzar”

Graph traversals

22

name: “Thomas Anderson”age: 29

name: “Morpheus”rank: “Captain”occupation: “Total badass”

name: “Trinity”

name: “Cypher”last name: “Reagan”

name: “Agent Smith”version: “1.0b”language: “C++”

name: “The Architect”

KNOWS

KNOWS

KNOWSKNOWS

KNOWSCODED BY

LOVES

disclosure: “secret”

disclosure: “public”


Graph traversals

import neo4jclass Friends(neo4j.Traversal): # Traversals ! queries in Neo4j

types = [ neo4j.Outgoing.KNOWS ]order = neo4j.BREADTH_FIRSTstop = neo4j.STOP_AT_END_OF_GRAPHreturnable = neo4j.RETURN_ALL_BUT_START_NODE

23



name: “Trinity”




KNOWS

KNOWS

KNOWSKNOWS

KNOWSCODED BY

LOVES



for friend_node in Friends(mr_anderson):print "%s (@ depth=%s)" % ( friend_node["name"],

friend_node.depth )


Graph traversals



23



name: “Trinity”




KNOWS

KNOWS

KNOWSKNOWS

KNOWSCODED BY

LOVES




friend_node.depth )


Graph traversals



23



name: “Trinity”




KNOWS

KNOWS

KNOWSKNOWS

KNOWSCODED BY

LOVES



Morpheus (@ depth=1)


friend_node.depth )


Graph traversals



23



name: “Trinity”




KNOWS

KNOWS

KNOWSKNOWS

KNOWSCODED BY

LOVES




Trinity (@ depth=1)


friend_node.depth )


Graph traversals



23



name: “Trinity”




KNOWS

KNOWS

KNOWSKNOWS

KNOWSCODED BY

LOVES




Trinity (@ depth=1)

Cypher (@ depth=2)


friend_node.depth )


Graph traversals



23



name: “Trinity”




KNOWS

KNOWS

KNOWSKNOWS

KNOWSCODED BY

LOVES




Trinity (@ depth=1)

Cypher (@ depth=2)

Agent Smith (@ depth=3)


friend_node.depth )


Graph traversals



23



name: “Trinity”




KNOWS

KNOWS

KNOWSKNOWS

KNOWSCODED BY

LOVES




Trinity (@ depth=1)

Cypher (@ depth=2)

Agent Smith (@ depth=3)


friend_node.depth )

Finding a place to start๏Traversals need a Node to start from

•QUESTION: How do I find the start Node?

•ANSWER: You use an Index

๏ Indexes in Neo4j are different from Indexes in Relational Databases

•RDBMSes use them for Joining

•Neo4j use them for simple lookup

24

IndexService index = getGraphDbIndexServiceSomehow();

Node mrAnderson = index.getSingleNode( "name", "Thomas Anderson" );

performTraversalFrom( mrAnderson );

Indexes in Neo4j๏The Graph *is* the main index

•Use relationship labels for navigation

•Build index structures *in the graph*

‣Search trees, tag clouds, geospatial indexes, et.c.

‣Linked/skip lists or other data structures in the graph

‣We have utility libraries for this

๏External indexes used *for lookup*

• Finding a (number of) points to start traversals from

•Major difference from RDBMS that use indexes for everything

25

A domain object implemented in Neo4jpublic interface Person {

String getName();void setName( String firstName, String lastName );

}

public final class PersonImpl implements Person {private final Node underlyingNode;public PersonImpl( Node underlyingNode ) {

this.underlyingNode = underlyingNode;}public String getName() {

return String.format("%s %s",underlyingNode.getProperty("first name"),underlyingNode.getProperty("last name") );

}public String setName(String firstName, String lastName) {

underlyingNode.setProperty("first name", firstName);underlyingNode.setProperty("last name", lastName);

}} 26

Neo4j as Software Transactional Memory๏ Implement objects as wrappers around Nodes and Relationships

•Neo4j is fast enough to allow you to read all state from the Node/Relationship

๏Mutating operations require transactions

•The changes are isolated from all other threads until committed

•Multiple mutations can be committed atomically

๏Nested transactions are flattened

•Makes it possible to have methods open their own transaction

๏Fits nicely with the OO paradigm

•More focus on data than on objects (comp. Object DBs) 27

Why not use an O/R mapper?๏Model evolution in ORMs is a hard problem

• virtually unsupported in most ORM systems

๏SQL is “compatible” across many RDBMSs

• data is still locked in

๏Each ORM maps object models differently

•Moving to another ORM == legacy schema support

‣except your legacy schema is a strange auto-generated one

๏Object/Graph Mapping is always done the same way

• allows you to keep your data through application changes

• or share data between multiple implementations 28

What an ORM doesn’t do

๏Deep traversals

๏Graph algorithms

๏Shortest path(s)

๏Routing

๏etc.

29

Path exists in social network๏Each person has on average 50 friends

30

Database # persons query timeRelational databaseNeo4j Graph DatabaseNeo4j Graph DatabaseRelational database

1 000 2 000 ms1 000 2 ms

1 000 000 2 ms1 000 000 way too long...

Tobias

Emil

JohanPeter

The performance impact in Neo4j depends only on the degree of each node. in an RDBMS it depends on the number of entries in the tables involved in the join(s).


30


1 000 2 000 ms1 000 2 ms

1 000 000 2 ms1 000 000 way too long...

Tobias

Emil

JohanPeter



30


1 000 2 000 ms1 000 2 ms

1 000 000 2 ms1 000 000 way too long...

Tobias

Emil

JohanPeter



30


1 000 2 000 ms1 000 2 ms

1 000 000 2 ms1 000 000 way too long...

Tobias

Emil

JohanPeter



30


1 000 2 000 ms1 000 2 ms

1 000 000 2 ms1 000 000 way too long...

Tobias

Emil

JohanPeter


๏20 million Nodes - represents places

๏62 million Edges - represents direct roads between places

•These edges have a length property, for the length of the road

๏Average optimal route, 100 separate roads, found in 100ms

๏Worst case route we could find:

•Optimal route is 5500 separate roads

•Total length ~770km

• Found in less than 3 seconds

๏Uses A* “best first” search

On-line real time routing with Neo4j

31

There’s a difference between least number of hops and least cost.

Routing with Neo4j - using Neo4j Graph-Algos# The cost evaluator - for choosing the best next nodeclass GeoCostEvaluator

include EstimateEvaluatordef getCost(node, goal)

straight_path_distance(node.getProperty("lat"), node.getProperty("lon"),goal.getProperty("lat"), goal.getProperty("lon") )

endend

# Instantiate the A* search functionpath_finder = AStar.new( Neo4j::instance,

RelationshipExpander.forTypes(DynamicRelationshipType.withName("road"),

Direction::BOTH ),DoubleEvaluator.new("length"), GeoCostEvaluator.new )

# Find the best path between New York City and San Franciscobest_path = path_finder.findSinglePath( NYC, SF )

32

Newest addition: Neo4j lets you REST๏Hello Neo4j REST server - Neo4j no longer needs to be embedded

๏Opens up Neo4j to your favorite platform (even if that isn’t Java)

• PHP, .NET, et.c. - libraries already exists!

• http://wiki.neo4j.org/content/Getting_Started_REST

๏Uses JSON for state transfer + browsable HTML for introspection

๏Atomic modification operations

๏Brand new declarative traversal framework

• Extensible using your favorite scripting language

‣javascript is included. Jython, JRuby, et.c. supported33

http://wiki.neo4j.org/content/Getting_Started_REST

http://wiki.neo4j.org/content/Getting_Started_REST

Other cool Graph Databases๏Sones GraphDB

•Graph Query Language - a SQL-like query language for graphs

๏Franz Inc. AllegroGraph

๏HypergraphDB

๏ InfoGrid

๏Twitter’s FlockDB

•Optimized for the Twitter use case - one level relationships

๏ Interestingly we all have different approaches34

http://www.sones.com/

http://www.sones.com/

http://www.franz.com/agraph/allegrograph/

http://www.franz.com/agraph/allegrograph/

http://www.kobrix.com/hgdb.jsp

http://www.kobrix.com/hgdb.jsp

http://infogrid.org/

http://infogrid.org/

http://github.com/twitter/flockdb

http://github.com/twitter/flockdb

One database to rule them all

35Image credits: The Lord of the Rings, New Line Cinema

Up until recently there was only one Database, the RDBMS. The days of a single database that rules all is over.

Use best suited storage for each kind of data

36Image credits: Unknown :’(

The era of using RDBMSes for all problems is over.Instead we should use the database most suited for the problem at hand.

Polyglot persistence

37

... we could even use multiple databases in conjunction, and let each database handle the things it does best.

{...}

{...}

{...}

Document


38

SQL && NOSQL

All databases are welcome!SQL and NOSQL - it is Not Only SQL!

{...}

{...}

{...}

Document

Finding out more

39

๏http://neo4j.org/ - project website

‣http://api.neo4j.org/ and http://components.neo4j.org/

‣http://wiki.neo4j.org/ - HowTos, Tutorials, Examples, FAQ, et.c.

‣http://planet.neo4j.org/ - aggregation of blogs about Neo4j

๏http://neotechnology.com/ - commercial licensing

๏http://twitter.com/neo4j/team - follow the Neo4j team

๏http://nosql.mypopescu.com/ - good source for news on NOSQLmonitors Neo4j and other NOSQL solutions

๏http://highscalability.com/ - has published a few articles about Neo4j

http://neo4j.org

http://neo4j.org

http://api.neo4j.org

http://api.neo4j.org

http://components.neo4j.org

http://components.neo4j.org

http://wiki.neo4j.org

http://wiki.neo4j.org

http://planet.neo4j.org

http://planet.neo4j.org



http://twitter.com/neo4j/team

http://twitter.com/neo4j/team

http://nosql.mypopescu.com

http://nosql.mypopescu.com

http://highscalability.com

http://highscalability.com

Buzzword summary

40

AGPLv3 ACID transactions

Embedded

NOSQL

Beer

A* routing

Open Source

Free Software

http://neo4j.org/

Software Transactional Memory

whiteboard friendly

Object mapping

Traversal Query language

SPARQL

Scaling to complexity

Shortest path

Semi structured

Schema free


Right tool for the right job

RESTful

GremlinIn-Graph indexes

http://neo4j.org

http://neo4j.org
