Adversarial Substructured Representation Learning for ...

Presenter: PengyangWangPengyang Wang, Yanjie Fu, Xiaolin Li, Hui Xiong

Adversarial Substructured Representation Learning for Mobile User Profiling

1

Outline

¨ BackgroundandMotivation¨ Definition and ProblemStatement¨ Methodology¨ Evaluation¨ Conclusion

Motivation Application: Toward Adaptive User Interfaces2

Mobileuserprofiling

Asimilaritygraphofusers,transportations,ODpairs

Adaptiveinterfacesby:(1) inferringtrippurposes,(2) transportmodes,(3) origin-destinationpairstoimproveuserengagementandperformances

Challenge I: Implicit User Patterns inMobile Activities3

¨ Humanactivities arespatially,temporally,andsociallystructural.

• Howcanweidentifyadatastructuretobetterdescribeamobileuser’sactivities?

From Users To Activity Graphs4

Spatial-temporaltransitionpatterns

UserActivityGraph

Office Hospital

Costco

Gas Station

MacDonaldWalmart

PreSchool

Auto Service

Home

Zoo

Sixflag

PizzaHut

RegalCinemas

Lab

Library IT

Department

Problem Formulation: Representation Learning with Activity Graphs5

f(,)=z

• Givenauserandcorrespondinguseractivitygraph,weaimtomaptheusertoaprofilevector

Office Hospital

Costco

Gas Station

MacDonaldWalmart

PreSchool

Auto Service

Home

Zoo

Sixflag

PizzaHut

RegalCinemas

Lab

Library IT

Department

UserProfileVector

Global Behavioral Pattern

¨ Entire structures: howauser’sactivitiesgloballyinteractwitheachother(stronglylink,weaklylink,nolink)

6

Office Hospital

Costco

Gas Station

MacDonaldWalmart

PreSchool

Auto Service

Home

Zoo

Sixflag

PizzaHut

RegalCinemas

Lab

Library IT

Department

Substructure Behavioral Pattern7

Office Hospital

Costco

Gas Station

MacDonaldWalmart

PreSchool

Auto Service

Home

Zoo

Sixflag

PizzaHut

RegalCinemas

Lab

Library IT

Department

¨ Substructures:topologyofsubgraphsthatfeaturetheuniquebehavioralpatternsofauser’sactivities

A Young Faculty with Young Kids

Circle:Personalized

Preference For A Close-loop

Consecutive Activity Pattern

High-frequencyVertexes: Personalized

Interests For ASpecific Type of

Activities

f(,, )=

Problem Reformulation: Representation Learning with Global and Sub-Structure Awareness8

Office

Lab

Library IT

Department

Office

PreSchool Home

UserProfileVector

Entire Structure Patterns SubstructurePatterns

Office Hospital

Costco

Gas Station

MacDonaldWalmart

PreSchool

Auto Service

Home

Zoo

Sixflag

PizzaHut

RegalCinemas

Lab

Library IT

Department

Preserving Entire-Structures9

Learned representation from hidden layer

Encode Decode

Preserving Substructures10

11

Preserving Substructures

12

Preserving Substructures

13

Preserving Substructures

Approximating Substructure Detector14

Encoder Decoder

SubstructureDetector

Discriminator+

-Real

Fake

1

0

Real

Fake

Deep First Search

Substructure

Substructure

xxDeep First

Search Substructure (Label)

ss

Training CNN

Pre-train aConvolutionalNeuralNetwork(CNN)toapproximatethetraditionalsubstructuredetector

Not differentiable

Approximating Substructure Detector14

Encoder Decoder

SubstructureDetector

Discriminator+

-Real

Fake

1

0

Real

Fake

Deep First Search

Substructure

Substructure

Not differentiable

xxDeep First

Search Substructure (Label)

ss

Training CNN

Pre-train aConvolutionalNeuralNetwork(CNN)toapproximatethetraditionalsubstructuredetector

Summary15

Encoder Decoder

Pre-trained CNN

xx xx

Substructure

ss

Substructure

ss

Discriminator+

-RealFake

1

0

Real

Fake

zz

• GeneratorAutoencoder linked with anapproximated substructuredetector (pre-trained CNN)

• DiscriminatorAmultilayerpercetron

• AdversarialTraining• Discriminatoraccuracy

• Generatorloss

Optimization16

¨ Training

¨ Testing

Discriminator Loss Generator Loss

Reconstruction Loss

Encoderxx

zz

Well Trained

What To Do Next: Inferring Next Activity Type17

User Representation

Next Activity Recommendation

User Profiling

User

Adversarial Substructured Learning

……

Office Hospital

Costco

Gas Station

MacDonaldWalmart

PreSchool

Auto Service

Home

Zoo

Sixflag

PizzaHut

RegalCinemas

Lab

Library IT

Department

Encoder Decoder

SubstructureDetector

Discriminator+

-Real

Fake

1

0

Real

Fake

SubstructureDetector

Substructure

Substructure

User Activity Graph

1. Givenatimeperiod,learnauser’sprofilesfromcorrespondinguseractivitygraph

2. Exploituserprofilestoforecastnextactivitytype

¨ Datao Mobileactivitycheckin data

ofNYCandTokyo

¨ EvaluationMetricso Theprecision@N ofactivity

categorypredictiono Theprecision@N ofnew

activityrecommendation¨ Baselines

o Autoencodero DeepWalk: usetruncated

randomwalkstolearnlatentrepresentations

o LINE: preservebothlocalandglobalnetworkstructureswithanedge-samplingalgorithm

o CNN: Convolutional NeuralNetwork

Overall Comparisons on New York and Tokyo Activity Check-in Data18

• Ourmodelachievesthebestperformancesonuserprofiling

• Substructures inagraphare essential for userbehaviorpatterns

Applythelearnedrepresentations topredictnextactivitytype(nextPOIcategory)

Study of Node and Circle Substructures19

¨ EvaluationMetricso Theprecision@N ofactivity

categoryprediction

o Theprecision@N ofnewactivityrecommendation

¨ Baselineso StructRL:considernodeand

circlesubstructureso StructRL-Node:only

considernodesubstructures

o StrucctRL-Circle:onlyconsidercirclesubstructure

¨ Findingso Circlesubstucture aremore

effectiveo Capturingmoresubgraph

topologiescanhelp

Conclusion

¨ Research Problem¨ Learn to profile users by both considering general interests and specific

interests for certain activity types

¨ Method¨ Users as Activity Graphs¨ Formulate modeling specific interests as preserving substructures of user

activity graphs¨ Propose an adversarial substructured learning model to integrate substructure

into representation learning

¨ Take Away Messages¨ Adversarial learning plays the role of regularization¨ Substructure is very important for quantifying user behavior patterns¨ Pre-train neural networks to approximate undifferentiable algorithms¨ Circle is more effective than independent vertexes for profiling users

20

18

Thanks!

Questions?

Will The Traditional Solution Work?24

Walmart

Hospital

Costco

Gas Station

PreSchool

Gym

Home

Zoo

Sixflag

RegalCinemas

T J MaxxALDI

Farmer Market

PetSmart

Pharmacy

Library

Office Hospital

Costco

Gas Station

MacDonaldWalmart

PreSchool

Auto Service

Home

Zoo

Sixflag

PizzaHut

RegalCinemas

Lab

Library IT

Department

Topologies,contents,locationsofsubgraphswilldynamicallychangeoverusers

Will The Traditional Solution Work?

0 0 1 1

0 0 1 1

0 0 0 0

0 0 0 0

25

0 0 0 0

0 0 0 0

0 1 1 0

0 1 1 0

Example:Dynamicbinaryindicatorofsubgraphsintheactivitymatrix/graphsoftwousers

Robustness Check15

@5 @10 @15 @20

Prec

isio

n@N

0.0

0.1

0.2

0.3

0.412 Apr. 2012 −− 12 Jun. 201213 Jun. 2012 −− 13 Aug. 201214 Aug. 2012 −− 14 Oct. 201215 Aug. 2012 −− 15 Oct. 201216 Oct. 2012 −− 16 Feb. 2013

@5 @10 @15 @20

Prec

isio

nnew@

N

0.00

0.05

0.10

0.15

0.2012 Apr. 2012 −− 12 Jun. 201213 Jun. 2012 −− 13 Aug. 201214 Aug. 2012 −− 14 Oct. 201215 Aug. 2012 −− 15 Oct. 201216 Oct. 2012 −− 16 Feb. 2013

@5 @10 @15 @20

Prec

isio

n@N

0.0

0.1

0.2

0.3

0.4

0.5Group 1Group 2Group 3

Group 4Group 5

@5 @10 @15 @20

Prec

isio

nnew@

N

0.00

0.05

0.10

0.15

0.20

0.25

0.3012 Apr. 2012 −− 12 Jun. 201213 Jun. 2012 −− 13 Aug. 201214 Aug. 2012 −− 14 Oct. 201215 Aug. 2012 −− 15 Oct. 201216 Oct. 2012 −− 16 Feb. 2013

New York

Tokyo

• Theperformancesofourmethodcanachieveasmallvarianceandarerelativelystable,especiallywhenKissmall.

¨ Five Periodso 12Apr.2012– 12Jun.2012o 13Jun.2012– 13Aug.2012o 14Aug.2012– 14Oct.2012o 15Aug.2012– 15Oct.2012o 16Oct.2012– 16Feb.2013

¨ Predictiono setthelastday’sactivitiesofeach

timeperiodasapredictivetarget