Predicting Dynamic Embedding Trajectory in Temporal ...srijan/pubs/jodie-kdd2019-slides.pdfPredicting Dynamic Embedding Trajectory in Temporal Interaction Networks Srijan Kumar Stanford

1

Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks

Srijan KumarStanford University

Georgia Institute of Technology

Jure LeskovecStanford University

Xikun ZhangUIUC

Code and Data: https://snap.stanford.edu/jodie

2

Temporal Interaction Networks

Time

[KDD’19]

Flexible way to represent time-evolving relations

Users Items

Feature

interaction user item time features

Represented as a sequence of interactions,

sorted by time:

3

Temporal Interaction Networks[KDD’19]

E-commerce Social media

Finance

WebEducation

IoT

Application domains Accounts Posts

…...

4

Temporal Interaction Networks[KDD’19]

E-commerce Social media

Finance

Web

Students Courses

Education

IoT

Application domains

…...

5

Problem Setup

Given a temporal interaction network

where

generate an embedding trajectory of every user

and an embedding trajectory of every item

[KDD’19]

interaction user item time features

6

Goal: Generate Dynamic Trajectory

Output: Dynamic trajectory in embedding space

Input: Temporal interaction network

[KDD’19]

1

2

4

3

56

7

ChallengesChallenges in modeling:• C1: How to learn inter-dependent user and item

embeddings? • C2: How to generate embedding for every point

in time?

Challenges in scalability: • C3: How to scalably train models on temporal

networks?

[KDD’19]

8

Existing Methods

Deep recommender systems• Time-LSTM (IJCAI 2017)• Recurrent Recommender Networks (WSDM

2017)• Latent Cross (WSDM 2018)

Dynamic co-evolution• Deep Coevolve (DLRS, 2016)

Temporal network embedding• CTDNE (BigNet, 2018)

Our model: JODIE

[KDD’19]

C1Co-

influence

C2Embed

any time

C3Train in batches

9

Our Model: JODIEJODIE: Joint Dynamic Interaction Embedding• Mutually-recursive recurrent neural network framework

[KDD’19]

ProjectionOperator

ProjectComponent

User RNN Item RNNUpdate Component

10

JODIE: Update Component[KDD’19]

User RNN Item RNN

f =

Weight matrices Ware trainable

• All users share the User-RNN parameters. Similar for items.

11

JODIE: Project Component

How can we predict the next item? • Rank items using distance in the embedding space

[KDD’19]

Projected embedding

Projection operatorTime Δ

Projected embedding

f =

User RNN Item RNN

12

Summary: JODIE Formulation

Update embeddings:

[KDD’19]

Loss:

Predicted next item is close to the real item

embeddingSmoothness in evolving

embeddings

Project user embedding:

Predict next item:

13

Challenges in Dynamic TrajectoriesChallenges in learning:• C1: How to learn inter-dependent user and item

embeddings? Solution: Update component• C2: How to generate embedding for every point in

time? Solution: Project component

Challenges in scalability: • C3: How to scalably train models on temporal

networks?

[KDD’19]

14

Standard Training Processes: N/ATraining must maintain temporal order

[KDD’19]

(1)

(2)

(3)

(4)

.

...

.

.

User 1

User 2

User 3

Split by user (or item): not allowed

Sequential processing: not scalable

1

2

3

4

4

32

1 5

6

Batch 1 Temporal inconsistency

15

T-batch: Temporal data batching algorithm

• Main idea: create each batch as an independent edge set

• Create a sequence of batches– Interactions in each batch are processed in

parallel– Process the batches in sequence to maintain

temporal ordering

[KDD’19]

T-batch: Batching for Scalability

16

T-batch: Batching for Scalability

Batch 2Batch 1 Batch 3

[KDD’19]

1

2

3

4

56

2

1

4

3

5

6

Iteratively select the maximal

independent edge set.

17

Challenges in Dynamic TrajectoriesChallenges in learning:• C1: How to learn inter-dependent user and item

embeddings? Solution: Update component• C2: How to generate embedding for every point in

time? Solution: Project component

Challenges in scalability: • C3: How to scalably train models on temporal

networks? Solution: T-batch Algorithm

[KDD’19]

18

Experiments: Prediction Tasks

• Temporal Link Prediction:–Which item i ∈ 𝐼 will user u interact with at

time t?• Temporal Node Classification:– Does a user u become anomalous after an

interaction?• Settings:– Temporal Splits: 80%, 10%, 10%–Metrics: Mean reciprocal rank, Recall@10,

AUROC

[KDD’19]

Code and Data: https://snap.stanford.edu/jodie

19

Datasets[KDD’19]

Dataset Users Items Interactions Temporal Anomalies

Reddit 10,000 984 672,447 366Wikipedia 8,227 1,000 157,474 217LastFM 980 1,000 1,293,103 -MOOC 7,047 97 411,749 4,066

NEW!

NEW!

Code and Data: https://snap.stanford.edu/jodie

20

Experiment 1: Link Prediction

JODIE outperforms baselines by > 20%

Mean Reciprocal

Rank

0.0

1.0

Latent Cross

0.42

0.18

Time-LSTM

0.60

RRN

0.73

0.39

0.17

CTDNE Deep Coevolve

JODIE

0.2

0.4

0.6

0.8

[KDD’19]

21

Experiment 2: Node Classification

JODIE outperforms all baselines by >12%

AUROC

0.5

1.0

Latent Cross

0.630.58

Time-LSTM

0.65

RRN

0.73

0.65 0.64

CTDNE Deep Coevolve

JODIE

0.6

0.7

0.8

0.9

[KDD’19]

22

Experiment 3: T-batch Speed-up

T-batch leads to 8.5x speed-up in training

5.1 minutes

44 minutes

JODIE without T-batch

JODIE with T-batch

Running Time

0

50

10

20

30

40

8.5x speed-up

[KDD’19]

23

Predicting Dynamic Embedding Trajectory in Temporal Interaction NetworksSrijan Kumar, Xikun Zhang, Jure Leskovec

Code and Data: https://snap.stanford.edu/jodie

JODIE generates and projects embedding

trajectories

• JODIE: a mutually-recursive RNN framework• T-batch: 8.5x training speed-up• Efficient in temporal link prediction and node classification• Extendible to > 2 entity types

24

Open Positions @ Georgia Tech

• Hiring multiple Ph.D. students• Research areas:–Machine Learning for Networks– Safety, Integrity, and Anti-Abuse– Computational Social Science

• Collaborations

Contact: [email protected]

25

Predicting Dynamic Embedding Trajectory in Temporal Interaction NetworksSrijan Kumar, Xikun Zhang, Jure Leskovec

Code and Data: https://snap.stanford.edu/jodie

JODIE generates and projects embedding

trajectories

• JODIE: a mutually-recursive RNN framework• T-batch: 8.5x training speed-up• Efficient in temporal link prediction and node classification• Extendible to > 2 entity types