Graph Database Overview and Feature Update Darren Wood Chief Architect, InfiniteGraph
May 12, 2015
Graph Database Overviewand Feature Update
Darren WoodChief Architect, InfiniteGraph
History
• Objectivity – Massively scalable, distributed object oriented database– Used in Government (DoD, Intelligence)
• Machine generated data such as sensor, acoustic…
– OEM Markets • Either complex data models, or high ingest or both
• Significant technical advantage in highly connected (many-to-many) data models
Copyright © InfiniteGraph
Graph Databases
• Key technical attributes• How Infinite Graph addresses these• Query and navigation• Challenges/Requirements of Distribution• Practical applications
Copyright © InfiniteGraph
Graph Databases
• Optimized around data relationships– Relationships as first class citizens– Super fast traversal between entities– Rich/flexible annotation of connections
• Small focused API (typically not SQL)– Natively work with concepts of Vertex/Edge– SQL has no concept of “navigation”– Most attempts based in SQL are convoluted
Copyright © InfiniteGraph
Distributed Graph Must Haves
• High performance distributed persistence• Ability to deal with remote data reads (fast)• Intelligent local cache of subgraphs• Distributed navigation processing• Distributed, multi-source concurrent ingest• Write modes supporting both strict and
eventual consistency
Copyright © InfiniteGraph
Some Code
Copyright © InfiniteGraph
Vertex alice = myGraph.addVertex(new Person(“Alice”)); Vertex bob = myGraph.addVertex(new Person(“Bob”)); Vertex carlos = myGraph.addVertex(new Person(“Carlos”)); Vertex charlie = myGraph.addVertex(new Person(“Charlie”));
alice.addEdge(new Meeting(“Denver”, “5-27-10”), bob);bob.addEdge(new Call(timestamp), carlos);carlos.addEdge(new Payment(100000.00), charlie);bob.addEdge(new Call(timestamp), charlie);
Alice Carlos CharlieBobMeets Calls Pays
Calls
Physical Storage Comparison
Copyright © InfiniteGraph
Meetings
P1 Place TimeP2Alice Denver 5-27-10Bob
Calls
From Time DurationToBob 13:20 25CarlosBob 17:10 15Charlie
Payments
From Date AmountToCarlos 5-12-10 100000Charlie
Met5-27-10Alice
Called13:20Bob
Payed100000Carlos
Charlie
Called17:10
Rows/Columns/Tables Relationship/Graph Optimized
Query and Navigation• Queries – but not as you know them• More like a rules based search and discovery• Asynchronous Results
Copyright © InfiniteGraph
Alice Carlos CharlieBobMeets Calls Pays
Calls
“Find all paths between Alice and Charlie”
“Find all paths between Alice and Charlie – within 2 degrees”
“Find all paths between Alice and Charlie – events in May 2010”
Navigation Example
Copyright © InfiniteGraph
// Create a qualifier that describes the target vertexQualifier findCharliePredicate =
new VertexPredicate(personType, "name == ’Charlie'");
// Construct a navigator which starts with Alice and uses a result qualifier// to find all paths in the graph to CharlieNavigator charlieFinder = alice.navigate(
Guide.SIMPLE_BREADTH_FIRST, // default guide Qualifier.ANY, // no path constraints
findCharliePredicate , // find paths ending with Charlie
myResultHandler); // fire results to supplied handler
// Start the navigatorcharlieFinder.start();
Management of Large Data Graphs
• Graphs grow quickly– Billions of phone calls / day in US– Emails, social media events, IP Traffic– Financial transactions
• Some analytics require navigation of large sections of the graph
• Each step (often) depends on the last• Must distribute data and go parallel
Copyright © InfiniteGraph
Basic Architecture
Copyright © InfiniteGraph
IG Core/API
ConfigurationNavigation Execution
Management Extensions
BlueprintsUser Apps
Objectivity/DB Distributed Database
Session / TX ManagementPlacement
Feature Update
Copyright © InfiniteGraph
2.0
Accelerated Ingest
Copyright © InfiniteGraph
IG Core/API
ConfigurationNavigation Execution
Management Extensions
Session / TX ManagementPlacement
Standard Blocking Ingest/Placement (MDP Plugin)
Objectivity/DB
App-1(Ingest V1)
App-2(Ingest V2)
App-3(Ingest V3)
V1V1 V2
V2 V3V3
App-1(E1 2{ V1V2})
App-2(E23{ V2V3})
App-3
E12E12 E23
E23
Accelerated Ingest
Copyright © InfiniteGraph
IG Core/API
ConfigurationNavigation Execution
Management Extensions
Session / TX Management
Placement(Standard)Placement
(Accelerated)
V1V1
V2V2
V3V3
E12E12
E23E23
Distributed
Pipelines
Sta
ging
Con
tain
ers P
ipeline Containers
E(1->2)
E(3->1)
E(2->3)
E(2->1)
E(2->3)E(3->1)
E(1->2)
E(3->2)
E(1->2)
E(2->3)
E(3->1)
E(2->1)
E(2->3)
E(3->1)
E(3->2)
E(1->2)
InfiniteGraph Visualizer
• Really nice flexible graph viewer• Browser style navigation and history• Full index support – search your data• Display connections around a selected point• Fully customize display to your data model • Full data view via selection
Copyright © InfiniteGraph
InfiniteGraph Visualizer
Copyright © InfiniteGraph
InfiniteGraph Visualizer
Copyright © InfiniteGraph
Indexing Framework
• Focused on providing choice !• Manual Indexes for grouping data• Automatic Indexes for cross population• Query interface with qualification language• Pluggable query operators• External index support (Lucene)
Copyright © InfiniteGraph
• Automated Distributed Navigation• Stored Loadable Navigators• Visualizer Navigation Plugins• More Visualizer Enhancements• More Import/Export support
Copyright © InfiniteGraph
>> next