S2Graph : A large-scale graph d atabase · Products that is interacted by users who interacted on products that I interact SELECT b.product_id, count(*) FROM user_products a, user_products

S2Graph : A large-scale graph database

with Hbase

daumkakao

2

Reference

1. HBase Conference 20151.http://www.slideshare.net/HBaseCon/use-cases-session-52.https://vimeo.com/128203919

2. Deview 20153. Apache Con BigData Europe

1.http://sched.co/3ztM4. Github: https://github.com/daumkakao/s2graph

http://www.slideshare.net/HBaseCon/use-cases-session-5

http://sched.co/3ztM

https://github.com/daumkakao/s2graph

3

Our Social Graph

Message

Writelength :

Read

Couponprice :

Presentprice : 3

affinity affinity:

affinity

affinity

affinity

affinity

affinity

affinity

affinity

Friend

Groupsize : 6

Emoticon

Eatrating :

Viewcount :

Playlevel: 6

Styleshare : 3

Advertise

Searchkeyword :

Listencount :

Likecount : 7

Comment

affinity

4

Our Social Graph

Messagelength : 9

Writelength : 3

affinity 6affinity: 9

affinity 3

affinity 3

affinity 4

affinity 1

affinity 2

affinity 2

affinity 9

Friend

Playlevel: 6

Styleshare : 3

Advertisectr : 0.32

Searchkeyword : “HBase"

Listencount : 6

Commentlength : 15

affinity 3

Message ID : 201

Ad ID : 603Music ID : 603

Item ID : 13

Post ID : 97

Game ID : 1984

5

Technical Challenges

1. Large social graph constantly changing

a. Scale

more than,social network: 10 billion edges, 200 million vertices, 50 million update on existing edges.user activities: over 1 billion new edges per day

6

Technical Challenges (cont)

2. Low latency for breadth first search traversal on connected data.

a. performance requirement

peak graph-traversing query per second: 20000response time: 100ms

7


3. Realtime update capabilities for viral effects

Person A

PostFast Person B

Comment

Person C

Sharing

Person D

MentionFast Fast

8


4. Support for Dynamic Ranking logic

a. Push strategy: Hard to change data ranking logic dynamically.

b. Pull strategy: Enables user to try out various data ranking logics.

9

Before

Each app server should know each DB’s sharding logic. Highly inter-connected architecture

Friend relationship SNS feeds Blog user activities Messaging

Messaging App

SNS App

Blog App

10

After

SNS App

Blog App

Messaging App

S2Graph DBstateless app servers

What is S2Graph?

daumkakao

12

What is S2Graph?

Storage-as-a-Service + Graph API = Realtime Breadth First Search

13

Example: Messanger Data Model

Participates

Chat Room

Message 1

Message 1

Message 1

Contains

Recent messages in my chat rooms.SELECT a.* FROM user_chat_rooms a, chat_room_messages b WHERE a.user_id = 1 AND a.chat_room_id = b.chat_room_id WHERE b.created_at >= yesterday

14

Example: Messanger Data Model

Participates

Chat Room

Message 1

Message 1

Message 1

Contains

Recent messages in my chat rooms.

curl -XPOST localhost:9000/graphs/getEdges -H 'Content-Type: Application/json' -d '

{

"srcVertices": [{"serviceName": "s2graph", "columnName": “user_id", "id":1}],

"steps": [

[{"label": "user_chat_rooms", "direction": "out", "limit": 100}], // step

[{"label": "chat_room_messages", "direction": "out", "limit": 10, “where”: “created_at >= yesterday”}]

]

}

'

15

Example: News Feed (cont)

FriendsPost1

Post 2

Post 3

create/like/share posts

Posts that my friends interacted.SELECT a.*, b.* FROM friends a, user_posts b WHERE a.user_id = b.user_id WHERE b.updated_at >= yesterday and b.action_type in (‘create’, ‘like’, ‘share’)

16

Example: News Feed (cont)

FriendsPost1

Post 2

Post 3

create/like/share posts

Posts that my friends interacted.


{


"steps": [

[{"label": "friends", "direction": "out", "limit": 100}], // step

[{"label": “user_posts", "direction": "out", "limit": 10, “where”: “created_at >= yesterday”}]

]

}

'

17

Example: Recommendation(User-based CF) (cont)

Similar UsersProduct 1

Product2

Product 3

user-product interaction(click/buy/like/share)

Products that similar user interact recently.SELECT a.* , b.* FROM similar_users a, user_products b WHERE a.sim_user_id = b.user_id AND b.updated_at >= yesterday

Batch

18

Example: Recommendation(User-based CF) (cont)

Products that similar user interact recently.curl -XPOST localhost:9000/graphs/getEdges -H 'Content-Type: Application/json' -d '

{

“filterOut”: {“srcVertices”: [{“serviceName”: “s2graph”, “columnName”: “user_id”, “id”: 1}],

“steps”: [[{“label”: “user_products_interact”}]]

},


"steps": [

[{"label": “similar_users", "direction": "out", "limit": 100, “where”: “similarity > 0.2”}], // step

[{"label": “user_products_interact”, "direction": "out", "limit": 10, “where”: “created_at >= yesterday and price >= 1000”}]

]

}

'

Similar UsersProduct 1

Product2

Product 3


Batch

19

Example: Recommendation(Item-based CF) (cont)

Similar Products

Product 1

Product2

Product 3


Product 1

Product 1

Product 1

Products that are similar to what I have interested.SELECT a.* , b.* FROM similar_ a, user_products b WHERE a.sim_user_id = b.user_id AND b.updated_at >= yesterday

Batch

20

Example: Recommendation(Item-based CF) (cont)

Products that are similar to what I have interested.


{


"steps": [

[{"label": “user_products_interact", "direction": "out", "limit": 100, “where”: “created_at >= yesterday and price >= 1000”}],

[{"label": “similar_products”, "direction": "out", "limit": 10, “where”: “similarity > 0.2”}]

]

}

'

Similar Products

Product 1

Product2

Product 3


Product 1

Product 1

Product 1

Batch

21

Example: Recommendation(Content + Most popular) (cont)

TopK(k=1) product per timeUnit(day)

Product1

Product2

Product 3


Daily top product per categories in products that I liked.SELECT c.*FROM user_products a, product_categories b, category_daily_top_products cWHERE a.user_id = 1 and a.product_id = b.product_id and b.category_id = c.category_id and c.time between (yesterday, today)

Category1

Category2

Product10

Product20

Product20

Today

Product10 Yesterday

Today

Yesterday

22

Example: Recommendation(Content + Most popular) (cont)

Daily top product per categories in products that I liked.curl -XPOST localhost:9000/graphs/getEdges -H 'Content-Type: Application/json' -d '

{


"steps": [


[{“label”: “product_cates”, “direction”: “out”, “limit”: 3}],

[{"label": “category_products_topK”, "direction": "out", "limit": 10]

]

}

'

TopK(k=1) product per timeUnit(day)

Product1

Product2

Product 3


Category1

Category2

Product10

Product20

Product20

Today

Product10 Yesterday

Today

Yesterday

23

Example: Recommendation(Spreading Activation) (cont)

Product 1

Product2

Product 3


Products that is interacted by users who interacted on products that I interactSELECT b.product_id, count(*)FROM user_products a, user_products bWHERE a.user_id = 1AND a.product_id = b.product_idGROUP BY b.product_id

24

Example: Recommendation(Spreading Activation) (cont)

Product 1

Product2

Product 3


Products that is interacted by users who interacted on products that I interactcurl -XPOST localhost:9000/graphs/getEdges -H 'Content-Type: Application/json' -d '

{


"steps": [


[{"label": “user_products_interact", "direction": "in", "limit": 10, “where”: “created_at >= today”}],

[{"label": “user_products_interact", "direction": "out", "limit": 10, “where”: “created_at >= 1 hour ago”}],

]

}

'

25

Realization

1. These examples resemble graphs.2. Object isVertex, Relationship is Edge.3. Necessary APIs: breadth first search on large scale graph.

26

S2Graph API: Vertex

Vertex:1. insert, delete, getVertex2. vertex id: what user

provided(string/int/long)

ID 1231-123Prop1 Val1Prop2 Val2

… …

27

S2Graph API: Edge

Edges:

1. Insert, delete, update, getEdge(like CRUD in RDBMS)

2. Edge reference: (from, to, label, direction)

3. Multiple props on edge.4. Every edges are ordered (details

follow).Edge Reference 1,101,”friend”,”out”

Prop1 Val1Prop2 Val2

… …

28

S2Graph API: indices

Degree Q1 Q2 Q3

1-friend-out-PK 3 c-103 b-102 a-101

1

101

102

103

Name: a

Name: b

Name: c

Ordered(DESC)

Indices:1. addIndex, createIndex2. Automatically keep edges ordered for

multiple indices.3. Support int/long/float/string data

types.

class Index { // define how to order edges. String indexName; List[Prop] indexProps;}

29

S2Graph API: Query

Query: getEdges, countEdges, removeEdges

Class Query {// Define breadth first search

List[VertexId] startVertices; List[Step] steps;}Class Step { // Define one breadth List[QueryParam] queryParams;}Class QueryParam { // Define each edges to traverse for current breadth String label; String direction; Map options;}

QueryParam

Step1 Step2

Query

30

What is S2Graph

Not support global computation(not like Apache Giraph, graphX). Not support graph algorithm like page rank, shortest path.

Storage-as-a-Service + Graph API = Realtime Breadth First Search

S2Graph is Not

31

Why S2Graph: Push vs Pull. Feeds with Push

1. Only timestamp can be used as scoring2. Hard to change scoring function dynamically

PostLike

Write(Fanout)Friends Feed Queue

Feed Queue

Feed Queue

Write # of friendsRead O(1) for friends

Storage AVG(# of friends) * total user activityQuery O(1)

32

1.Different weights to different action types: Like = 0.8, Click = 0.1…2.Client can change scoring dynamically.

PostLike

Friends

Why S2Graph: Push vs Pull. Feeds with Pull

Write O(1)Read None

Storage total user activityQuery O(1) for friends + O(# of friends)

33

Pull >> push only if

1. fast response time: 10 ~ 100ms2. throughput: 10K ~ 20K QPS

S2Graph provide linear scalability on

1. number of machine.2. bfs search space(how many edges that single query will traverse).

more detail on benchmark section later.

Why S2Graph: S2Graph Supports Pull + Push

34

Why S2Graph: Simplify Data Flow

S2Graph

Write API + Query DSL

WAL log

OpenSourced

User/Item Similarity

Apache Spark (Batch Computing Layer)

TopK Counter Others

S2Graph Bulk Loader

will be open sourced soon

35

Why S2Graph: Built in A/B test

1. Register Query Template: Each Query template have impressionId.2. Insert Click/Impression event into S2Graph as Edge insert.

36

Why S2Graph: Just Insert Edge

S2Graph

1. user activity history. 2. friends feed.3. user-item based collaborative filtering.4. topK ranking(most popular, segmented most popular).

and many many more.just think your service as graph model.

S2Graph Internal

daumkakao

38

How to store the data - EdgeLogical View

a. Fetching an edge between two specific vertexb. Lookup Table to reach indexed edges for update, increment, delete operations

1. Snapshot edges : Up-to-date status of edge

Tgt Vertex ID1 Tgt Vertex ID2 Tgt Vertex ID3

Src Vertex ID1 Properties Properties Properties

Src Vertex ID2 Properties Properties Properties

2. Indexed edges : Edges with index

Index Values | Tgt Vertex ID1 Index Values | Tgt Vertex ID2

Src Vertex ID1 Non-index Properties Non-index Properties

a. Fetches edges originating from a certain vertex in order of index

39

How to store the data - VertexLogical View

1. Vertex : Up-to-date status of Vertex

columnrow

Property Key1 Property Key2

Src Vertex ID1 Value1 Value2

Vertex ID2 Value1 Value2

40

Problem

Update/Delete edge is hard.

- it is not feasible to traverse all edges to find edges to delete.- indexedEdge is ordered, so in worst case, client need to fetch all edges to find edge to delete.- this make delete/update edge become O(N).

Backtracking from snapshotEdge- read snapshot edge, then random access to indexed edge to delete. O(1)

Problem- need atomicity on update snapshot edge and indexed edge.- this require atomic operation(transaction) on multiple rows on HBase.- some times partial failure on this update yield broken states, such as zombie edges.

41

How to update edge

IndexedEdge Degree Q1 Q2 Q3

1-friend-out-PK 3 c-103 b-102 a-101

age:30, gender:M age:21 age:15, gender:F

SnapshotEdge Degree Q1 Q2 Q3

1-friend-out-PK 3 103 102 101

name:c:t0 age:30:t0

gender:M:t0name:b:t0 age:21:t0

name:a:t0 age:15:t0

gender:F:t0

curl -XPOST localhost:9000/graphs/edges/insert -H ‘Content-Type: Application/json’ -d ‘[ {“timestamp”: t0, “from”: 1, “to”: 101, “label”: “friend”, “props”: {“name”: “a”, “age”: 15, “gender”: “F”}}, {“timestamp”: t0, “from”: 1, “to”: 102, “label”: “friend”, “props”: {“name”: “b”, “age”: 21}}, {“timestamp”: t0, “from”: 1, “to”: 103, “label”: “friend”, “props”: {“name”: “c”, “age”: 30, “gender”: “M”}]‘

42


1-friend-out-PK 3 103 102 101

name:c:t0 age:30:t0

gender:M:t0

name:b:t0 age:21:t0

name:a:t0 name:d:t1 age:15:t0 age:26:t1

gender:F:t0

IndexedEdge: delete(1, (a-101)) insert(1, (d-101))

curl -XPOST localhost:9000/graphs/edges/update -H ‘Content-Type: Application/json’ -d ‘[ {“timestamp”: t-1, “from”: 1, “to”: 101, “label”: “friend”, “props”: {“name”: “k”, “age”: -10}} {“timestamp”: t1, “from”: 1, “to”: 101, “label”: “friend”, “props”: {“name”: “d”, “age”: 26}}]‘

How to update edge (cont)

1.Fetch SnapshotEdge2.check pending mutations and

retry 3.Build Update on Snapshot/

Indexed Edge4.CAS on new SnapshotEdge5.Mutate indexedEdge6.CAS on new SnapshotEdge

if pending mutations exist, other thread mutate this, so commit pending mutations and retry. If CAS is failed at 4, other thread lock this, so retry.


1-friend-out-PK 3 103 102 101name:c:t0 age:30:t0

gender:M:t0name:b:t0 age:21:t0

name:a:t0 age:15:t0

gender:F:t0

43

IndexedEdge Degree Q0 Q1 Q2 Q3

1-friend-out-PK

3 d-101 c-103 b-102 a-101

age:26,gender:F age:30, gender:M age:21 age:15, gender:F


1-friend-out-PK 3 103 102 101

name:c:t0 age:30:t0

gender:M:t0

name:b:t0 age:21:t0

name:a:t0 name:d:t1 age:15:t0 age:26:t1

gender:F:t0

IndexedEdge: delete(1, (a-101)) insert(1, (d-101))

1.Fetch SnapshotEdge2.Apply mutations stored in

SnapshotEdge if exist3.Build Update on Snapshot/


If any failure exist on 5, abort and retry from 1. it is safe to issue same mutation multiple time since s2graph is idempotent.


44


1-friend-out-PK 3 d-101 c-103 b-102

age:26,gender:F age:30, gender:M age:21


1-friend-out-PK 3 103 102 101

name:c:t0 age:30:t0

gender:M:t0

name:b:t0 age:21:t0

name:d:t1 age:26:t1

gender:F:t01.Fetch SnapshotEdge2.Apply mutations stored in

SnapshotEdge if exist3.Build Update on Snapshot/


If CAS is failed at 6, retry from 1


45

1.Fetch SnapshotEdge2.Build Update on Snapshot/



1-friend-out-PK 3 d-101 c-103 b-102

age:26,gender:F age:30, gender:M age:21


1-friend-out-PK 3 103 102 101

name:c:t0 age:30:t0

gender:M:t0

name:b:t0 age:21:t0

name:d:t1 age:26:t1

gender:F:t0


46

Summary

No atomic update between SnapshotEdge and IndexedEdge. - this can yield failure on update/delete on edge, and this is problem.

Then should we use transaction?

Pros: solve our problem. no failure on update/delete edge.Cons: need one more read even though there is no contention.

The probability of contention on edge is very low. stick to our retry logic with CAS.

Benchmarks

daumkakao

48

HBase Table Configuration

1. setDurability(Durability.ASYNC_WAL)

2. setCompressionType(Compression.Algorithm.LZ4)

3. setBloomFilterType(BloomType.Row)

4. setDataBlockEncoding(DataBlockEncoding.FAST_DIFF)

5. setBlockSize(32768)

6. setBlockCacheEnabled(true)

7. pre-split by (Intger.MaxValue / regionCount). regionCount = 120 when create table(on 20 region server).

49

HBase Cluster Configuration

• each machine: 8core, 32G memory, SSD

• hfile.block.cache.size: 0.6

• hbase.hregion.memstore.flush.size: 128MB

• otherwise use default value from CDH 5.3.1

• s2graph rest server: 4core, 16G memory

50

Performance

1. Total # of Edges: 100,000,000,000(100,000,000 row x 1000 column)2. Test environment

a. Zookeeper server: 3b. HBase Masterserver: 2c. HBase Regionserver: 20d. App server: 4 core, 16GB Rame. Write traffic: 5K / second

51

- Benchmark Query : src.out(“friend”).limit(100).out(“friend”).limit(10)- Total concurrency: 20 * # of app server

Performance

2. Linear scalability

Late

ncy

0

50

100

150

200

QPS

0

1,000

2,000

3,000

4,000

# of app server1 2 4 8

QPS(Query Per Second) Latency(ms)

46454543

3,491

1,763

88546443 45 45 46

# of app server1 2 3 4 5 6 7 8

500

1000

1500

2000

2500

3000

QPS

Performance

3. Varying width of traverse (tested with a single server)

Late

ncy

0

87.5

175

262.5

350

QPS

0

500

1,000

1,500

2,000

Limit on first step20 40 80 200 400 800

QPS Latency(ms)

327

164

84351911 61122237

570

1,023

1,821

11 19 3584

164

327

- Benchmark Query : src.out(“friend”).limit(x).out(“friend”).limit(10)- Total concurrency = 20 * 1(# of app server)

53

- All query touch 1000 edges.- each step` limit is on x axis.- Can expect performance with given query`s search space.

Performance

4. Different query path(different I/O pattern)

Late

ncy

0

37.5

75

112.5

150

QPS

0

80

160

240

320

400

limits on path10 -> 100 100 -> 10 10 -> 10 -> 10 2 -> 5 -> 10 -> 10 2 -> 5 -> 2 -> 5 -> 10

QPS Latency(ms)

3234362314

307.5292.1274.4

435.3695

14 23 36 34 32

54

Performance

5. Write throughput per operation on single app server

Insert operation

Late

ncy

0

1.25

2.5

3.75

5

Request per second

8000 16000 800000

55

Performance

6. write throughput per operation on single app server

Update(increment/update/delete) operation

Late

ncy

0

2

4

6

8

Request per second

2000 4000 6000

56

Stats

1. HBase cluster per IDC (2 IDC)- 3 Zookeeper Server- 2 HBase Master- (20 + 40) HBase Slave

2. App server per IDC- 10 server for write-only- 30 server for query only

3. Real traffic- read: 10K ~ 20K request per second

- now mostly 2 step queries with limit 100 on first step.- write: over 5k ~ 10k request per second

57

58

Through S2Graph !

59

Now Available As an Open Source- https://github.com/daumkakao/s2graph- Finding contributors and mentors

Contact- Doyoung Yoon : [email protected]

https://github.com/daumkakao/s2graph

mailto:[email protected]

S2Graph : A large-scale graph d atabase · Products that is interacted by users who interacted on products that I interact SELECT b.product_id, count(*) FROM user_products a, user_products

Documents