GraphConnect Europe 2016 - NoSQL Polyglot Persistence: Tools and Integrations - William Lyon

NoSQL Polyglot PersistenceNeo4j Tools and Integrations

William Lyon @lyonwj

William LyonDeveloper Relations @neo4j

[email protected] @lyonwj lyonwj.com

Does Neo4j work with ______ ?

Languages, frameworks, databases…

Does Neo4j work with ______ ?

Languages

Does Neo4j work with ______ ?

Frameworks

Does Neo4j work with ______ ?

Databases

Does Neo4j work with ______ ?

Deployment

Does Neo4j work with ______ ?

Frameworks

Does Neo4j work with ______ ?

Languages, frameworks, databases…

Polyglot Persistence

DataStorageandBusinessRulesExecu5on

DataMiningandAggrega5on

Applica'on

GraphDatabaseCluster

Neo4j Neo4j Neo4j

AdHocAnalysis

BulkAnaly'cInfrastructureHadoop,EDW…

DataScien'st

EndUser

DatabasesRela5onalNoSQLHadoop

Polyglot Persistence

• Different types of data in different ways• Take advantage of strengths of different databases

http://martinfowler.com/bliki/PolyglotPersistence.html

Polyglot Persistence

Functionality Database type

Shopping Cart Rapid session reads / writes

Key-value store

Orders / Product Catalog

Frequent reads Document

Customer social graph

Recommendation Graph

http://www.jamesserra.com/archive/2015/07/what-is-polyglot-persistence/

Added Complexity

Challenge #1: Data Sync

MIGRATEALLDATA

MIGRATESUBSET

DUPLICATESUBSET

Non-GraphQueries GraphQueries

GraphQueriesNon-GraphQueries

AllQueries

Rela3onalDatabase

GraphDatabase

Application

Application

Application

NonGraphData

AllData

?

Course CatalogBrowse & Search

• Good use-case for a document database

?

Tool #1: Neo4j Doc Manager

• Automatically sync documents from MongoDB to Neo4j• Convert BSON documents into graph model• Initial version available• Feedback welcome!

http://neo4j.com/developer/mongodb/

Neo4j Doc ManagerAutomatically sync documents from MongoDB to Neo4j

Document to property graph{ "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics": [ "keynote", "spring" ], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "Juergen Hoeller", "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" }}

{ "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics": [ "keynote", "spring" ], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "Juergen Hoeller", "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" }}

Document to property graph

https://github.com/neo4j-contrib/neo4j_doc_manager

Neo4j + MongoDB Webinar

http://neo4j.com/webinars/

Building a polyglot application with Neo4j and MongoDB

http://neo4j.com/blog/couchbase-jdbc-integrations-neo4j-3-0/

Couchbase + Neo4j integration

Challenge #2: Operations

http://neo4j.com/developer/docker/

https://hub.docker.com/_/neo4j/

docker pull neo4j

Tool #2: Official Neo4j Docker Image

https://www.docker.com/products/docker-compose

docker-compose.yml

Challenge #3: Data Models

Tool #3: Neo4j Cassandra Data Import Tool

http://neo4j.com/developer/cassandra/

• Python command line app• Uses Python Neo4j and Cassandra drivers

Neo4j Cassandra Data Import ToolCassandra schema metadata

python connector.py parse -k playlist

Tables become nodes

Artist

Track

Column family to property graph

What’s that?

Neo4j Cassandra Data Import ToolUser defined data model mappings

Neo4j Cassandra Data Import Tool

{p}, node property{r}, relationship{u}, unique constraint field{i}, create index

User defined data model mappings

Neo4j Cassandra Data Import ToolUser defined data model mappings

Neo4j Cassandra Data Import ToolGenerates LOAD CSV Cypher

python connector.py export -k playlist -t track_by_id,artists_by_first_letter

Neo4j 3.0

Neo4j 3.0

Bolt Stored ProceduresDrivers

Neo4j 3.0 Stored procedures

User-defined procedures are written in Java, deployed into the database, and called from Cypher.

http://neo4j.com/docs/java-reference/current/#procedures

https://github.com/neo4j-contrib/neo4j-apoc-procedures

https://github.com/neo4j-contrib/neo4j-apoc-procedures/blob/master/readme.adoc

Northwind

Northwind

Bolt and Neo4j 3.0 Drivers

Binary protocol for Neo4j

http://neo4j.com/docs/developer-manual/current/#driver

https://github.com/jexp/neo4j-spark-connector

Neo4j Spark ConnectorUsing Neo4j 3.0 Java Driver with Bolt

https://github.com/johnymontana/neovis.js

neovis.jsEmbeddable graph visualization powered by Neo4j 3.0 Bolt Javascript driver

Resources

http://neo4j.com/developer/

Neo4j Developer Resources

Questions?

@lyonwj [email protected]