Learning Binary Code for Personalized Fashion Recommendationopenaccess.thecvf.com/content_CVPR_2019/papers/Lu... · 2019-06-10 · Learning Binary Code for Personalized Fashion Recommendation

Learning Binary Code for Personalized Fashion Recommendation

Zhi Ludagger Yang Husect Yunchao Jiangdagger Yan Chensect Bing Zengsect

School of Information and Communication Engineering

University of Electronic Science and Technology of Chinadaggerzhilujiangyunchaostduestceducn sectyanghueecyaneezenguestceducn

Abstract

With the rapid growth of fashion-focused social networks

and online shopping intelligent fashion recommendation is

now in great needs Recommending fashion outfits each

of which is composed of multiple interacted clothing and

accessories is relatively new to the field The problem be-

comes even more interesting and challenging when consid-

ering usersrsquo personalized fashion style Another challenge

in a large-scale fashion outfit recommendation system is the

efficiency issue of itemoutfit search and storage In this pa-

per we propose to learn binary code for efficient personal-

ized fashion outfits recommendation Our system consists

of three components a feature network for content extrac-

tion a set of type-dependent hashing modules to learn bi-

nary codes and a matching block that conducts pairwise

matching The whole framework is trained in an end-to-

end manner We collect outfit data together with user label

information from a fashion-focused social website for the

personalized recommendation task Extensive experiments

on our datasets show that the proposed framework outper-

forms the state-of-the-art methods significantly even with a

simple backbone

1 Introduction

Fashion-focused social networks have become a vibrant

realm where millions of individuals share and post daily

fashion-related activities A huge amount of fashion outfits

have been created by users in these communities Mining

desirable fashion outfits from this massive data set is very

challenging but critical to the development of these online

fashion communities Therefore there is a great need for

intelligent fashion recommendation techniques The num-

ber of possible outfits grows exponentially with the number

of items in each garment category The storage complexity

and the retrieval efficiency of the outfits are essential for the

The corresponding author is Yang Hu This work was supported by

the National Natural Science Foundation of China (61602090) and the 111

Project (B17008)

Figure 1 Examples of recommended outfits for three users where

the outfits in red boxes are user-created ones

deployment of a fashion recommendation system in prac-

tice which however have not been well addressed before

Existing related works can be loosely classified into two

types based on the way they evaluate how compatible the

items in an outfit are The first type of approaches model

pairwise compatibilities between fashion items Veit et

al [28] propose to use a Siamese network to learn the dis-

tance between paired items Hu et al [7] learn a functional

factorization and compute the compatibility based on pair-

wise inner product Vasileva et al [27] learn type-aware

embedding for fashion items and use a fully-connected

layer as a generalized distance function for compatibility

prediction The second type of approaches seek to model

high order relations among the items of an outfit Li et

al [12] use a recurrent neural network to predict set com-

patibility Han et al [3] treat outfits as a sequence of items

and compute the compatibility score through LSTM

The number of items in a fashion inventory is usually

very large and the number of outfits that can be composed

by these items is orders of magnitude larger To deploy

a practical fashion recommendation system efficiency be-

10562

comes an extremely important problem The items and out-

fits should be stored in an economical way The compati-

bility of an outfit should be evaluated not only accurately

but also efficiently And the most compatible items and

outfits need to be retrieved swiftly These are issues that

havenrsquot been well handled by previous works In this work

we take them into consideration and explore the hash tech-

nique for fashion recommendation Learning to hash has

been extensively studies for efficient image retrieval [15]

Some recent works have also combine it with collabora-

tive filtering algorithms to recommend individual items to

users [35 19 32 13] We incorporate it into the task of sets

composition problem for personalized outfit recommenda-

tion

We design a neural network for efficient personalized

fashion outfit recommendation The network captures the

favors of different users and learn the compatibility between

fashion items It is composed of three components A fea-

ture network is first used to extract content features Then a

set of type-dependent hashing modules convert the features

and user taste representations into binary codes Finally

the overall preference of a user to an outfit is computed by

a matching block that conducts pairwise weighted hashing

matching Both visual and textual information are utilized

in our model Since existing datasets are either two small

or lacking user information we collect a new dataset which

contains more than 138000 outfits created by hundreds of

online users We evaluate our method on this large scale

dataset Extensive experiments show that it outperforms the

state-of-the-art methods even with a simple backbone and

binary representations

2 Related Work

21 Fashion recommendation

Fashion analysis [17] such as clothing recognition [20]

latent embedding [24 5 10] parsing [2 31 14] re-

trieval [18 14] and recommendation [16 8 21 28 6] have

attracted many attentions in recent years Among the plenty

of works that studied fashion related problems we focus on

those that related to the recommendation task in the follow-

ing

Liu et al [16] introduce a latent SVM based model for

occasion-oriented clothing recommendation McAuley el

al [21] propose a parametric distance transformation to

learn the item compatibility Veit et al [28] utilize Siamese

network to learn embedding for fashion items These works

do not consider the composition of multiple items in an out-

fit and only focus on the matching between two items Some

methods seek to directly model the high-order relationships

between items Li et al [12] apply multi-modality fusion

and RNN-based multi-instance pooling models to classify

the outfit quality Han et al train a bidirectional LSTM as

a scorer to predict the compatibility of an outfit Vasileva et

al [27] learn type-aware mapping for different kinds of

items and utilize metric layers to learn the compatibility

As for personalized composition problem Hu et al [7]

make an initial exploration A functional tensor factoriza-

tion method is proposed to model the user-item and item-

item interactions in multiple latent spaces Nevertheless

they use hand-crafted features and do not jointly optimize

the representation of images Hsiao el al [6] propose a sub-

set selection model for selecting a minimal set of garments

that maximize the compatibility and versatility which in-

troduce a new recommendation topic However those men-

tioned models do not consider the efficiency of the system

22 Learning to hash

Learning to hash is the task of learning a compact bi-

nary code for the input item It aims to maintain the near-

est neighbor relation of the original space in the hamming

space Hashing methods have become a promising and pop-

ular technique for efficient similarity search which also re-

duce the storage cost of data The basic idea in learning to

hash is similarity preserving [29] ie minimizing the gap

between the similarity computed in hash-coded space and

the similarity in the original space Most existing hashing

methods first introduce some relaxations to their problems

by learning real-valued embedding and then take the sign of

the values to obtain binary codes [15 36 11] which how-

ever often suffer from quantization loss Recently Cao et

al [1] proposed a method to learn hash codes by continua-

tion with convergence guarantees

There have been some works reported that apply hashing

technique to the recommendation problem [35 13] Zhou et

al [35] learn binary code that preserves the preference of

users to items in collaborative filtering Lian et al [13] pro-

pose a discrete content-aware matrix factorization model

However these methods cannot be applied directly to the

outfit composition problem since they only recommend sin-

gle items while an outfit contains multiple interacted items

In this work we model outfit compatibility through pair-

wise interactions and employ the weighted hashing tech-

nique [34 30 33] for matching users and items

3 The Proposed Approach

31 Problem formulation

Suppose there are N fashion categories (eg top bot-

tom and shoes) The number of items in the n-th cate-

gory is donated by Ln and the number of users is U Let

X(n) = x(n)1 x

(n)Ln

donate all items in the n-th cate-

gory where x(n)i is the i-th item in it Then an outfit with N

items with each from one category can be represented as

Oi = x(1)i1

x(2)i2

x(N)iN

(1)

10563

Fashion Outfit

Visual

Embedding

Type-dependent

Encoder

Binary Codes

Shared Parameters

User

Embedding

Sign Activation

Approximated Sign

Λ(i)

ltlatexit sha1_base64=rfCku9ci3K7vgkxe24EjhtGUciU=gtAAAB9HicbVC7TsMwFL3mWcqrwMhiUSGVpUpYYKMSCwNDkehDakPlOE5r1XGC7VQqUb+DhQGEWJn4EjZGgT3MUDLkSwdnXOu7vXxE8G1cZwvtLS8srq2ntvIb25t7+wW9vbrOk4VZTUai1g1faKZ4JLVDDeCNRPFSOQL1vD7l2OMWBK81jemmHCvIh0JQ85JcZKXvvaRgNyl5X4yahTKDplZwK8SNwZKV58PHxXAKDaKXy2g5imEZOGCqJ1y3US42VEGU4FG+XbqWYJoX3SZS1LJYmY9rLJ0SN8bJUAh7GyTxo8UX9PZCTSehj5NhkR09Pz3lj8z2ulJjz3Mi6T1DBJp4vCVGAT43EDOOCKUSOGlhCquL0V0x5RhBrbU96W4M5eZHUT8uu5TduseLAFDk4hCMogQtnUIErqEINKNzDIzzDCxqgJSK3qbRJTSbOYAQO8jFSUIg==ltlatexitgtltlatexit sha1_base64=svZhBVbsHHrBvZFrEgY3JMixr9Y=gtAAAB9HicbVA9TwJBFHyHX4hfqKXNRmKCDbmz0ZLExsICEwETOMne3gIb9vbO3Xck5MLvsLHQGFtjJ3xgWuUHCSTSYz8LeTpBIYdB1v53C2vrG5lZxu7Szu7dUD48apk41Yw3WSxjRBQw6VQvIkCJX9INKdRIHk7GF3PPaYayNidY+ThPsRHSjRF4yilfzurY2G9DGrivNpr1xxa+4cZJV4OalAjkavNUNY5ZGXCGT1JiO5yboZ1SjYJJPS93U8ISyER3wjqWKRtz42fzoKTmzSkj6sbZPIZmrvycyGhkziQKbjCgOzbI3EzOin2rxMqCRFrthiUT+VBGMya4CEQnOGcmIJZVrYWwkbUk0Z2p5KtgRv+curpHVR8yy8yp1N6+jCCdwClXw4BLqcAMNaAKDJ3iGV3hzxs6L8+58LKIFJ585hj9wPn8AGGqRlA==ltlatexitgt

Λ(i)


Λ(u)

ltlatexit sha1_base64=J8nrVDOoasICnvBK9JuY3a+HWTY=gtAAAB9HicbVC7TsMwFL3mWcqrwMhiUSGVpUpYYKMSCwNDkehDakPlOE5r1XGC7VQqUb+DhQGEWJn4EjZGgT3MUDLkSwdnXOu7vXxE8G1cZwvtLS8srq2ntvIb25t7+wW9vbrOk4VZTUai1g1faKZ4JLVDDeCNRPFSOQL1vD7l2OMWBK81jemmHCvIh0JQ85JcZKXvvaRgNyl5XSk1GnUHTKzgR4kbgzUrz4ePiuAEC1UhsBzFNIyYNFUTrluskxsuIMpwKNsq3U80SQvuky1qWShIx7WWTo0f42CoBDmNlnzR4ov6eyEik9TDybTIipqfnvbH4n9dKTXjuZVwmqWGSTheFqcAmxuMGcMAVo0YMLSFUcXsrpj2iCDW2p7wtwZ38iKpn5Zdy2cYsWBKXJwCEdQAhfOoAJXUIUaULiHR3iGFzRAT+gVvU2jS2g2cwBgN5AJ6clC4=ltlatexitgtltlatexit sha1_base64=Xl6d6MLbtUfRtruAo3XBjLwpMQk=gtAAAB9HicbVA9TwJBFHyHX4hfqKXNRmKCDbmz0ZLExsICEwETOMm7vQU27O2du3sk5MLvsLHQGFtjJ3xgWuUHCSTSYz8LeTpAIro3rfjuFtfWNza3idmlnd2oHx41NJxqihr0ljE6iFAzQSXrGm4EewhUQyjQLB2MLqe+e0xU5rH8t5MEuZHOJC8zykaKndWxsN8TGrpufTXrni1tw5yCrxclKBHI1e+asbxjSNmDRUoNYdz02Mn6EynAo2LXVTzRKkIxywjqUSI6b9bH70lJxZJST9WNknDZmrvycyjLSeRIFNRmiGetmbif95ndT0ryMyyQ1TNLFon4qiInJrAEScsWoERNLkCpubyV0iAqpsT2VbAne8pdXSeui5ll+51Xqbl5HEU7gFKrgwSXU4QYa0AQKTAMrDmjJ0X5935WEQLTj5zDHgfP4AKrKRoA==ltlatexitgt

Λ(u)


Matching Block

ruOtltlatexit sha1_base64=Xf316HlsuGvu8VLLPSScDNok5bw=gtAAACC3icbVDLSsNAFL2pr1pfsS7dhBbBhZTEjS4LbtxZwT6gKWEymbRDJ5MwMxFKyFbc+A6A25cKOLWH3Dn3zhpu9DWC8MczrmXe+7xE0alsu1vo7Syura+Ud6sbG3v7O6Z+9WOjFOBSRvHLBY9H0nCKCdtRRUjvUQQFPmMdP3xRaF3b4mQNOY3apKQQYSGnIYUI6Upz6wJL0tP3AipEUYsu8q9zPVjFshJpL9M5XnumXW7YULWgbOHNSb1ae7RwBoeeaXG8Q4jQhXmCEp+46dqEGGhKKYkbzippIkCIRkPQ15CgicpBNb8mtI80EVhgLbiypuzviQxFsjCnOwvTclEryP+0fqrC80FGeZIqwvFsUZgyS8VWEYwVUEGwYhMNEBZUe7XwCAmElY6vokNwFk9eBp3ThqPxtVNv2jCrMhxCDY7BgTNowiW0oA0Y7uEZXuHNeDBejHfjY9ZaMuYzBCnjM8ft1ueQA==ltlatexitgtltlatexit sha1_base64=850APKZpvsYnf2RrocRBVGDIY=gtAAACC3icbVC7TsMwFHXKq5RXgJHFaoXEgKqEBcZKLGwUiT6kJoocx2mtOnZkO0hVlJ2FX2FhACFWfoCNv8FpM0DLlSwfnXOv7rknTBlV2nG+rdra+sbmVn27sbO7t39gHx71lcgkJj0smJDDECnCKCc9TTUjw1QSlISMDMLpdakPHohUVPB7PUuJn6AxpzHFSBsqsJsyyLNzL0F6ghHLb4sg90LBIjVLzJfroigCu+W0nXnBVeBWoAWq6gb2lxcJnCWEa8yQUiPXSbWfI6kpZqRoeJkiKcJTNCYjAzlKiPLz+S0FPDVMBGMhzeMaztnfEzlKVGnOdJam1bJWkv9po0zHV35OeZppwvFiUZwxqAUsg4ERlQRrNjMAYUmNV4gnSCKsTXwNE4K7fPIq6F+0XYPv3FbHqeKogxPQBGfABZegA25AFQABogGbyCN+vJerHerY9Fa82qZo7Bn7I+fwDimJwnltlatexitgt


EncoderFeature

CNN+1

ltlatexit sha1_base64=K7oPXV0Y0WikxykHk7TBQR595TQ=gtAAAB6XicbZA9SwNBEIbn4leMX1FLm8UgCEK4s9HOgI1lFPMByRH2NnPJkr29Y3dPCEf+gY2FIoKV8jOP2LtXpJCE19YeHjfGXZmgkRwbVz3yymsrK6tbxQ3S1vbO7t75f2Dpo5TxbDBYhGrdkA1Ci6xYbgR2E4U0igQ2ApG13neekCleSzvzThBP6IDyUPOqLHW3ZnXK1fcqjsVWQZvDpWr73fIVe+VP7v9mKURSsME1brjuYnxM6oMZwInpW6qMaFsRAfYsShphNrPppNOyIl1+iSMlX3SkKn7uyOjkdbjKLCVETVDvZjl5n9ZJzXhpZ9xmaQGJZt9FKaCmJjka5M+V8iMGFugTHE7K2FDqigz9jglewRvceVlaJ5XPcu3XqXmwkxFOIJjOAUPLqAGN1CHBjAI4RGe4cUZOUOqM2Ky04855D+CPn4wdR9Y6nltlatexitgtltlatexit sha1_base64=4b7opZVfaUuobKz5QqEbQXF78hg=gtAAAB6XicbZBNSwMxEIYn9avWr6pHL8EiCELZ9aLHghePVWwttEvJptk2NJtdklmhLP0HXjwo4tV5M1Y9ruQVtfCDy8M0Nm3jBV0qLnfZPS2vrG5lZ5u7Kzu7dUD08atskM1y0eKIS0wmZFUpq0UKJSnRSI1gcKvEYjm9m9ccnYaxM9ANOUhHEbKhlJDlDZ91f+P1qzat7c9FV8AuoQaFmvrVGyQ8i4VGrpi1Xd9LMciZQcmVmFZ6mRUp42M2FF2HmsXCBvl80yk9c86ARolxTyOdu78nchZbO4lD1xkzHNnl2sz8r9bNMLoOcqnTDIXmi4+iTFFM6OxsOpBGcFQTB4wb6XalfMQM4+jCqbgQOWTV6F9Wfcd3m1hlfEUYYTOIVz8OEKGnALTWgBhwie4RXeyJi8kHfysWgtkWLmGP6IfP4A2oCM1Q==ltlatexitgt

+1ltlatexit sha1_base64=K7oPXV0Y0WikxykHk7TBQR595TQ=gtAAAB6XicbZA9SwNBEIbn4leMX1FLm8UgCEK4s9HOgI1lFPMByRH2NnPJkr29Y3dPCEf+gY2FIoKV8jOP2LtXpJCE19YeHjfGXZmgkRwbVz3yymsrK6tbxQ3S1vbO7t75f2Dpo5TxbDBYhGrdkA1Ci6xYbgR2E4U0igQ2ApG13neekCleSzvzThBP6IDyUPOqLHW3ZnXK1fcqjsVWQZvDpWr73fIVe+VP7v9mKURSsME1brjuYnxM6oMZwInpW6qMaFsRAfYsShphNrPppNOyIl1+iSMlX3SkKn7uyOjkdbjKLCVETVDvZjl5n9ZJzXhpZ9xmaQGJZt9FKaCmJjka5M+V8iMGFugTHE7K2FDqigz9jglewRvceVlaJ5XPcu3XqXmwkxFOIJjOAUPLqAGN1CHBjAI4RGe4cUZOUOqM2Ky04855D+CPn4wdR9Y6nltlatexitgtltlatexit sha1_base64=4b7opZVfaUuobKz5QqEbQXF78hg=gtAAAB6XicbZBNSwMxEIYn9avWr6pHL8EiCELZ9aLHghePVWwttEvJptk2NJtdklmhLP0HXjwo4tV5M1Y9ruQVtfCDy8M0Nm3jBV0qLnfZPS2vrG5lZ5u7Kzu7dUD08atskM1y0eKIS0wmZFUpq0UKJSnRSI1gcKvEYjm9m9ccnYaxM9ANOUhHEbKhlJDlDZ91f+P1qzat7c9FV8AuoQaFmvrVGyQ8i4VGrpi1Xd9LMciZQcmVmFZ6mRUp42M2FF2HmsXCBvl80yk9c86ARolxTyOdu78nchZbO4lD1xkzHNnl2sz8r9bNMLoOcqnTDIXmi4+iTFFM6OxsOpBGcFQTB4wb6XalfMQM4+jCqbgQOWTV6F9Wfcd3m1hlfEUYYTOIVz8OEKGnALTWgBhwie4RXeyJi8kHfysWgtkWLmGP6IfP4A2oCM1Q==ltlatexitgt

minus1ltlatexit sha1_base64=uKOU6x64OBS5PdQY7gwQ1Q8V47o=gtAAAB6XicbZA9SwNBEIbn4leMX1FLm8Ug2BjubLQzYGMZxXxAcoS9zVyyZGv2N0TwpFYGOhiGDlP7Lzj1i7l6TQxBcWHt53hp2ZIBFcG9f9cgorq2vrG8XN0tb2zu5eef+gqeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLRdZ63HlBpHst7M07Qj+hA8pAzaqx1d+b1yhW36k5FlsGbQ+Xq+x1y1Xvlz24ZmmE0jBBte54bmL8jCrDmcBJqZtqTCgb0QF2LEoaofaz6aQTcmKdPgljZZ80ZOr+7shopPU4CmxlRM1QL2a5+VWSU146WdcJqlByWYfhakgJib52qTPFTIjxhYoU9zOStiQKsqMPU7JHsFbXHkZmudVzKtV6m5MFMRjuAYTsGDC6jBDdShAQxCeIRneHFGzpPz6rzNSgvOvOcQsj5+AFU46pltlatexitgtltlatexit sha1_base64=cwxTvFVIYQpYkt5mbKb+W59BA=gtAAAB6XicbZBNSwMxEIYn9avWr6pHL8EieLHsetFjwYvHKrYW2qVk02wbms0uyaxQlv4DLx4U8eo8uaMW33oK0vBB7emSEzb5gqadHzvklpbX1jc6u8XdnZ3dsqB4etW2SGS5aPFGJ6YTMCiW1aKFEJTqpESwOlXgMxzez+uOTMFYm+gEnqQhiNtQykpyhs+4vH615tW9uegq+AXUoFCzX3qDRKexUIjV8zaru+lGOTMoORKTCu9zIqU8TEbiq5DzWJhg3y+6ZSeOWdAo8S4p5HO3d8TOYutncSh64wZjuxybWb+V+tmGF0HudRphkLzxUdRpigmdHY2HUgjOKqJA8aNdLtSPmKGcXThVFwIvLJq9C+rPuO7xawyviKMMJnMI5+HAFDbiFJrSAQwTP8ApvZExeyDv5WLSWSDFzDH9EPn8A3YqM1w==ltlatexitgt










-

+

b(3)i3

isin minus1+1Dltlatexit sha1_base64=LRH82Oqn3alVMMHmMzBDvOcQyQ=gtAAAE0XicbVNLb9QwEHbbBUp5tXCDAxZRpVZ9aJM9lGNFqwoJLSp9bCsl2ZXjeFurjrO1HegqjYSQOPEPuHHmCnf+Bv+GcbJdmm0tJRqPv2ms2cmGgiuTbP5d2p6pnHn7r3Z+3MPHj56GR+4WlHp5mi7JCmIlXHEdFMcMkODTeCHQ8UI0kk2FF0tmXPjz4ypXkqD8xwwMKEnEje55QYcPXmXwadPCq6+VJruejlvNcqAi6DfM1dXXGDorvdm3ea681y4ZuGOzKczZUf2PzfGO3tzDzPYhTmiVMGiqI1r7bHJgwJ8pwKlgxF2SaDQg9IyfMB1OShOkwL29S4EXwxLifKvikwaX3OiMnidbDJAJkQsypnjyzzvHZIt43uE3UEO+lJMaWlwpdRjenLFUMFBEBSZJBZpjCmnImKavF1CaBACoeBa2rN3XYc6lZUtaie9nApsU26fGMVeMGjEEg1DFIRump0QRCslAHchj55nNiImMYWPmAsk+gZwE9nmgmSl8N8yD3IFSFJOH54Xvhb4M89zxip67Gog4NXq13AEDCIv4faqSOk+CpyR6EPjS8YLLMXhHpRGTPNO30PoVz6qxlr8TjlmciXgCbF1lEoAF9tmiyMK7VpvlbMGLC3ax1rJtcWweezFqkD+1jjXHiPiNhI0vJgg7Y1IwDpgUtsuus44qO5itUELVNAa4N0IIUQEBTtjBjsuDpQa7ayaDhR3Mm6nYkGfxVW7XUlvE6P4RR3ci8i6t+iYYdQoBVlqdrVAy453nIZb7+zXQeDoBbV0j798pq+jCREPHwT0LmGZ3cnZvGh1v3W2tex9grN+gas2iF+gVWkIu2kCb6C3aRYeIoqoJqFfjf2G8PG58aXCjo9NeI8Q7XV+PYPRSSc5Q==ltlatexitgt

EncoderFeature

CNNminus1

ltlatexit sha1_base64=uKOU6x64OBS5PdQY7gwQ1Q8V47o=gtAAAB6XicbZA9SwNBEIbn4leMX1FLm8Ug2BjubLQzYGMZxXxAcoS9zVyyZGv2N0TwpFYGOhiGDlP7Lzj1i7l6TQxBcWHt53hp2ZIBFcG9f9cgorq2vrG8XN0tb2zu5eef+gqeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLRdZ63HlBpHst7M07Qj+hA8pAzaqx1d+b1yhW36k5FlsGbQ+Xq+x1y1Xvlz24ZmmE0jBBte54bmL8jCrDmcBJqZtqTCgb0QF2LEoaofaz6aQTcmKdPgljZZ80ZOr+7shopPU4CmxlRM1QL2a5+VWSU146WdcJqlByWYfhakgJib52qTPFTIjxhYoU9zOStiQKsqMPU7JHsFbXHkZmudVzKtV6m5MFMRjuAYTsGDC6jBDdShAQxCeIRneHFGzpPz6rzNSgvOvOcQsj5+AFU46pltlatexitgtltlatexit sha1_base64=cwxTvFVIYQpYkt5mbKb+W59BA=gtAAAB6XicbZBNSwMxEIYn9avWr6pHL8EieLHsetFjwYvHKrYW2qVk02wbms0uyaxQlv4DLx4U8eo8uaMW33oK0vBB7emSEzb5gqadHzvklpbX1jc6u8XdnZ3dsqB4etW2SGS5aPFGJ6YTMCiW1aKFEJTqpESwOlXgMxzez+uOTMFYm+gEnqQhiNtQykpyhs+4vH615tW9uegq+AXUoFCzX3qDRKexUIjV8zaru+lGOTMoORKTCu9zIqU8TEbiq5DzWJhg3y+6ZSeOWdAo8S4p5HO3d8TOYutncSh64wZjuxybWb+V+tmGF0HudRphkLzxUdRpigmdHY2HUgjOKqJA8aNdLtSPmKGcXThVFwIvLJq9C+rPuO7xawyviKMMJnMI5+HAFDbiFJrSAQwTP8ApvZExeyDv5WLSWSDFzDH9EPn8A3YqM1w==ltlatexitgt












-

+

b(2)i2

isin minus1+1Dltlatexit sha1_base64=XEz8tYvmw0LQd6lL3sl3N9s5H8=gtAAAE0XicbVNLb9QwEHbpAqW8WrjBAYuoUlEf2qSHcqxoVSGhRaWPbaUku3Icp7XqOFvbga7SSAiJE+AG2eucOdv8G8YJ9ul2dZSovH4+2Y+e2aigeDatNtp25Nt27fuTtzbb+g4ePHsNP+nqLFeUHdBMZOooIpoJLtmB4Uawo4FiJI0EO4xON+354UemNMkvhkOWJiSY8kTTokBV3uRdAtorJXLHqvyn7B+14ZcBkUK+7ykhuUva3+nNNebVcLXzfckeFsLP3Ap9n6zv9+envQZzRPGXSUEG09t32wIQFUYZTwcrZINdsQOgpOWY+mJKkTIdFdZMSL4Anxkmm4JMGV96rjIKkWgTCJApMSd68sw6x2cLeMgDlFDvJuRGFteJnQV3ZywTDFQRAQkSQe5YQprypmkrBFTmxQCqHgUtKneJKDgkvLlrQWn+QCmwzbp8YxV4waMQSDUMUhG6YnRBEKyUAdyGNnuc2IiYxhY2YDyT6BnBT2RaCZKX03LILCgVKUk4dnpe+FvgyLwvHKvrsciDgzernaAQMICh9ptImT4KnInoQ+MLxgosxeFtlEZM81zfQkppn1VjL3w7HLM5EPAG2rioJwAL7bFFk4T2rzXI24cUFO19JrnNtMWwee7E6kL85zrXLiLiJBA0vJki7IxKw9pnUtouuMvbru1ht0AI1tAF4N0IIEUHBTpnBjosDpUY7q6YLxZ2M261Z0Gdx3W6X0jvEKH7eBHf+i4iSGzRsEwq0sipVp37ARQfm1Mbb6241weAobVLpP171X+t7MFEQ8fBPUuYZndydq8bXWVXVv1PsBYv0H1mkHP0Uu0iFy0jjbQW7SDDhBFX9FP9Av9bu21hq3PrS819NbUiPMUNVbr2z88ZZzjltlatexitgt

EncoderFeature

CNNminus1













-

+

b(1)i1

isin minus1+1Dltlatexit sha1_base64=vrZRrKcqp6K3CjmXDwp1M8IADCI=gtAAAE0XicbVNNT9wwEDWwbSn9gvbWHmp1hQQCVptwoEdUEKpUbUX5WJCS7MpxHLBwnMV2WlYhUlWpp6D3nrutb33bTfdJwsW7JgKdF4N7Ms2cmHAiuTbv9d2p6pnHn7r3Z+3MPHj56GR+4WlXp5mi7JCmIlXHIdFMcMkODTeCHQ8UI0ko2FF4tmXPjz4ypXkqD8xwwIKEnEgec0oMuPrzL1uHha9fMlZLvo57zuFz6WfrzmrK45f9Lb78812q10ufNNwRkZzc+UH9v4839jtL8x896OUZgmThgqitee0BybIiTKcClbM+ZlmA0LPyAnzwJQkYTrIy5sUeBE8EY5TBZ80uPReZ+Qk0XqYhIBMiDnVk2fWOT5bxPsGd4ga4r2URNjyUqHL6OaUpYqBIiIgSTLIDFNYU84kZbWY2iQQQEWjoHX1Jn4d5FxatqSV+DgT2KTYPjWOuGLUiCEYhCoO2TA9JYpQSAbqQB47z2xGTGQEGzPnSYJ5CSwz33NTOE5Qe7nTShFMXl4Xnhu4Mkgz5tu0XdWfRGlRq+WO2AAYRGT1VS50nwlEQXAl82Xf9yDN5RacgkzQttLjiWTXW8naCMYszEU2AratMAjDfPlsYWnjParOcLXhxwS7W4ptcWwybx16sCuRtjXPtMSJuI0HDiwnS3ogErAMmte2i64yD6i5WG7RABa0B3o0QQoRQsDNmcNPBvlKjnVXTheJOxu1WLOizqGq3K+kdYhSqIM70WE8S0adggFWlGWqlM94FLTXS7j7Xe362BwFN76FdL+3fKXvRgoqHj4J4FTLMzObs3ja7bctZb7gcY6zeoWrPoBXqFlpCDNtAmeot20SGi6Cv6iX6h3439xrDxufGlgk5PjTjPUG01vv0DM6ac4Q==ltlatexitgt

yltlatexit sha1_base64=8595zeR0Lu8S+v8tKGJJ2mP3rY=gtAAAB6HicbZC7SwNBEMbn4iuJr6ilzWIQrMKdjZYBG8sEzAOSEPb25pI1e3vH7p5wHKktbCwUsRPJDvGzePQhMWPjxfTPszPiJ4Nq47rdT2Njc2t4plsq7esHh5Wj47aOU8WwxWIRq65PNQousWW4EdhNFNLIF9jxJzezvPOASvNY3pkswUFER5KHnFFjrWY2rFTdmjsXWQdvCdV66TH4AIDGsPLVD2KWRigNE1TrnucmZpBTZTgTOC33U40JZRM6wp5FSSPUg3w+6JScWycgYazsk4bM3d8dOY20ziLfVkbUjPVqNjPy3qpCa8HOZdJalCyxUdhKoiJyWxrEnCFzIjMAmWK21kJG1NFmbG3KdsjeKsrr0P7suZZbnrVugsLFeEUzuACPLiCOtxCA1rAAOEJXuDVuXeenTfnfVFacJY9JBHzucPWP+OvA==ltlatexitgtltlatexit sha1_base64=eTz41HquW0A6JDnITJ63IQVpJw=gtAAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSLZY8OKxBfsBbSib7aRdu9mE3Y0QQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvXXZKW9s7u3vlcrB4dHxSfX0rKvjVDHssFjEqh9QjYJL7BhuBPYThTQKBPaC2d2i3ntCpXksH0yWoBRieQhZ9RYq52NqjW37i5FNsEroAaFWqPq13AcszRCaZigWg88NzF+TpXhTOC8Mkw1JpTN6AQHFiWNUPv5ctE5ubLOmISxsk8asnRT+Q00jqLAtsZUTPV67WF+V9tkJqw4edcJqlByVYfhakgJiaLq8mYK2RGZBYoU9zuStiUKsqMzaZiQDWT96E7k3ds9z2ak23iKMMF3AJ1+DBLTThHlrQAQYIzAKb86j8+K8Ox+r1pJTzJzDHzmfP+FljOg=ltlatexitgt


xltlatexit sha1_base64=Rkz9c4p4XAkmN0KphOFyrYjZnY=gtAAAB6HicbZC7SgNBFIbPxluMt6ilzWAQrMKujXYGbCwTMBdIljA7OZuMmZ1dZmbFuOQJbCwUsfUZfBI7S9EyaXQxB8GPv7HOacEySCa+O6X05uZXVtfSOWdja3tndK+4fNHScKoZ1FotYtQKqUXCJdcONwFaikEaBwGYwvJrkzTtUmsfyxowS9CPalzzkjBpr1e67xZJbdqciy+DNoXT58fBdAYBqtjZ6cUsjVAaJqjWbc9NjJ9RZTgTOC50Uo0JZUPax7ZFSSPUfjYddExOrNMjYazsk4ZM3d8dGY20HkWBrYyoGejFbGL+l7VTE174GZdJalCy2UdhKoiJyWRr0uMKmREjC5QpbmclbEAVZcbepmCP4C2uvAyNs7JnueaWKi7MlIcjOIZT8OAcKnANVagDA4RHeIYX59Z5cl6dt1lpzpn3HMIfOe8U4qPdA==ltlatexitgtltlatexit sha1_base64=7rvijkQYNKVvRpzSybqYSJxAazk=gtAAAB6HicbZBNT8JAEIan+IX4hXr0spGYeCKtFz2SePEIiQUSaMh2mcLKdtvsbo2k4Rd48aAxXv1J3vw3LtCDgm+yyZN3ZrIzb5gKro3rfjuljc2t7Z3ybmVv+DwqHp80tZJphj6LBGJ6oZUo+ASfcONwG6qkMahwE44uZ3XO4+oNEkvZmmGMR0JHnEGTXWaj0NqjW37i5E1sEroAaFmoPqV3+YsCxGaZigWvc8NzVBTpXhTOCs0s80ppRN6Ah7FiWNUQf5YtEZubDOkESJsk8asnBT+Q01noah7YzpmasV2tz879aLzPRTZBzmWYGJVt+FGWCmITMryZDrpAZMbVAmeJ2V8LGVFFmbDYVG4K3evI6tKqnuWWW2u4RRxlOINzuAQPrqEBd9AEHxggPMMrvDkPzovz7nwsW0tOMXMKf+R8gDfkYzmltlatexitgt






tanh(10x)ltlatexit sha1_base64=OuIriLuVsmBekh2zW+UIJKEQPZ0=gtAAAB8XicbZA9TwJBEIbn8AvxC7W02UhMsCF3NFqS2FhiIh8RLmRv2YMNe3uX3TkjufAvbCw0xtZY+ecYErFHyTTZ68M5OdeYNECoOu++0UNja3tneKu6W9YPDoLxSdvEqWa8xWIZ625ADZdC8RYKlLybaE6jQPJOMLmZ1zuPXBsRq3ucJtyP6EiJUDCK1nroI1Xjquc+XQ7KFbfmLkTWwcuhArmagJXfxizNOIKmaTG9Dw3QT+jGgWTfFbqp4YnlE3oiPcsKhpx42eLjWfkwjpDEsbaPoVk4f6eyGhkzDQKbGdEcWxWa3Pzv1ovxfDaz4RKUuSKLT8KU0kwJvPzyVBozlBOLVCmhd2VsDHVlKENqWRD8FZPXod2veZZvqtXGm4eRxHO4Byq4MEVNOAWmtACBgqe4RXeHOO8OOOx7K14OQzpBHzucPXWWP+w==ltlatexitgtltlatexit sha1_base64=OuIriLuVsmBekh2zW+UIJKEQPZ0=gtAAAB8XicbZA9TwJBEIbn8AvxC7W02UhMsCF3NFqS2FhiIh8RLmRv2YMNe3uX3TkjufAvbCw0xtZY+ecYErFHyTTZ68M5OdeYNECoOu++0UNja3tneKu6W9YPDoLxSdvEqWa8xWIZ625ADZdC8RYKlLybaE6jQPJOMLmZ1zuPXBsRq3ucJtyP6EiJUDCK1nroI1Xjquc+XQ7KFbfmLkTWwcuhArmagJXfxizNOIKmaTG9Dw3QT+jGgWTfFbqp4YnlE3oiPcsKhpx42eLjWfkwjpDEsbaPoVk4f6eyGhkzDQKbGdEcWxWa3Pzv1ovxfDaz4RKUuSKLT8KU0kwJvPzyVBozlBOLVCmhd2VsDHVlKENqWRD8FZPXod2veZZvqtXGm4eRxHO4Byq4MEVNOAWmtACBgqe4RXeHOO8OOOx7K14OQzpBHzucPXWWP+w==ltlatexitgt


tanh(x)ltlatexit sha1_base64=Aveb5mCQMZ96t8TU6SHIBa6QQXU=gtAAAB73icbZA9TwJBEIbn8AvxC7W02UhMsCF3NFqS2FhiIh8JXMjesgcb9vbO3TkjufAnbCw0xtaY+ecYErFHyTTZ68M5OdeYNECoOu++0UNja3tneKu6W9YPDoLxSdvEqWa8xWIZ625ADZdC8RYKlLybaE6jQPJOMLmZ1zuPXBsRq3ucJtyP6EiJUDCK1ur2kapx9elyUK64NXchsg5eDhXI1RyUvrDmKURV8gkNabnuQn6GdUomOSzUj81PKFsQke8Z1HRiBsW+w7IxfWGZIw1vYpJAv390RGI2OmUWA7I4pjs1qbmVeimG134mVJIiV2z5UZhKgjGZH0+GQnOGcmqBMi3sroSNqaYMbUQlG4K3evI6tOs1zJdvdJw8ziKcAbnUAUPrqABt9CEFjCQ8Ayv8OY8OCOuOxbC04+cwpJHz+QN7G4+Gltlatexitgtltlatexit sha1_base64=Aveb5mCQMZ96t8TU6SHIBa6QQXU=gtAAAB73icbZA9TwJBEIbn8AvxC7W02UhMsCF3NFqS2FhiIh8JXMjesgcb9vbO3TkjufAnbCw0xtaY+ecYErFHyTTZ68M5OdeYNECoOu++0UNja3tneKu6W9YPDoLxSdvEqWa8xWIZ625ADZdC8RYKlLybaE6jQPJOMLmZ1zuPXBsRq3ucJtyP6EiJUDCK1ur2kapx9elyUK64NXchsg5eDhXI1RyUvrDmKURV8gkNabnuQn6GdUomOSzUj81PKFsQke8Z1HRiBsW+w7IxfWGZIw1vYpJAv390RGI2OmUWA7I4pjs1qbmVeimG134mVJIiV2z5UZhKgjGZH0+GQnOGcmqBMi3sroSNqaYMbUQlG4K3evI6tOs1zJdvdJw8ziKcAbnUAUPrqABt9CEFjCQ8Ayv8OY8OCOuOxbC04+cwpJHz+QN7G4+Gltlatexitgt













+1ltlatexit sha1_base64=K7oPXV0Y0WikxykHk7TBQR595TQ=gtAAAB6XicbZA9SwNBEIbn4leMX1FLm8UgCEK4s9HOgI1lFPMByRH2NnPJkr29Y3dPCEf+gY2FIoKV8jOP2LtXpJCE19YeHjfGXZmgkRwbVz3yymsrK6tbxQ3S1vbO7t75f2Dpo5TxbDBYhGrdkA1Ci6xYbgR2E4U0igQ2ApG13neekCleSzvzThBP6IDyUPOqLHW3ZnXK1fcqjsVWQZvDpWr73fIVe+VP7v9mKURSsME1brjuYnxM6oMZwInpW6qMaFsRAfYsShphNrPppNOyIl1+iSMlX3SkKn7uyOjkdbjKLCVETVDvZjl5n9ZJzXhpZ9xmaQGJZt9FKaCmJjka5M+V8iMGFugTHE7K2FDqigz9jglewRvceVlaJ5XPcu3XqXmwkxFOIJjOAUPLqAGN1CHBjAI4RGe4cUZOUOqM2Ky04855D+CPn4wdR9Y6nltlatexitgtltlatexit sha1_base64=4b7opZVfaUuobKz5QqEbQXF78hg=gtAAAB6XicbZBNSwMxEIYn9avWr6pHL8EiCELZ9aLHghePVWwttEvJptk2NJtdklmhLP0HXjwo4tV5M1Y9ruQVtfCDy8M0Nm3jBV0qLnfZPS2vrG5lZ5u7Kzu7dUD08atskM1y0eKIS0wmZFUpq0UKJSnRSI1gcKvEYjm9m9ccnYaxM9ANOUhHEbKhlJDlDZ91f+P1qzat7c9FV8AuoQaFmvrVGyQ8i4VGrpi1Xd9LMciZQcmVmFZ6mRUp42M2FF2HmsXCBvl80yk9c86ARolxTyOdu78nchZbO4lD1xkzHNnl2sz8r9bNMLoOcqnTDIXmi4+iTFFM6OxsOpBGcFQTB4wb6XalfMQM4+jCqbgQOWTV6F9Wfcd3m1hlfEUYYTOIVz8OEKGnALTWgBhwie4RXeyJi8kHfysWgtkWLmGP6IfP4A2oCM1Q==ltlatexitgt1

ltlatexit sha1_base64=+YxSaEU5ps68FCvfSFWLLN0wI=gtAAAB6HicbZC7SgNBFIbPxluMt6ilzWAQrMKujekM2FgmYC6QLGF2cjYZMzu7zMwKcckT2FgoYusz+CR2lr6Jk0uhiT8MfPzOcw5J0gE18Z1v5zc2vrG5lZ+u7Czu7dUDw8auo4VQwbLBaxagdUo+ASG4Ybge1EIY0Cga1gdD3NWeoNIlrRkn6Ed0IHnIGTXWqnu9YsktuzORVfAWULr6ePiuAkCtVzs9mOWRigNE1Trjucmxs+oMpwJnBS6qcaEshEdYMeipBFqP5sNOiFn1umTMFb2SUNm7u+OjEZaj6PAVkbUDPVyNjXyzqpCSt+xmWSGpRslGYCmJiMt2a9LlCZsTYAmWK21kJG1JFmbG3KdgjeMsrr0LzouxZrnulqgtz5eEETuEcPLiEKtxADRrAAOERnuHFuXOenFfnbV6acxY9xBHzvsP6C+PLg==ltlatexitgtltlatexit sha1_base64=G80TKlOggVS1bdO2UARKLbBlog=gtAAAB6HicbZBNSwMxEIYnftb6VfXoJVgET2XXix4LXjy2YD+gXUo2nW1js9klyQpl6Sw4kERr4kb4b03YP2vpC4OGdGTLzhqkUxnreN9nY3Nre2S3tlfcPDo+OKyenbZNkmmOLJzLR3ZAZlEJhyworsZtqZHEosRNO7ub1zhNqIxL1YKcpBjEbKREJzqyzmv6gUvVq3kJ0HfwCqlCoMah89YcJz2JUlktmTM3UhvkTFvBJc7KcxgyviEjbDnULEYTZAvFp3RS+cMaZRo95SlCf3RM5iY6Zx6DpjZsdmtTY36v1MhvdBrlQaWZR8eVHUSapTej8ajoUGrmVUweMa+F2pXzMNOPWZVN2IfirJ69D+7rmO2761bpXxFGCc7iAKDhBupwDw1oAQeEZ3iFNJIXsg7+Vi2bpBi5gz+iHz+AHRFjKA=ltlatexitgt

1ltlatexit sha1_base64=+YxSaEU5ps68FCvfSFWLLN0wI=gtAAAB6HicbZC7SgNBFIbPxluMt6ilzWAQrMKujekM2FgmYC6QLGF2cjYZMzu7zMwKcckT2FgoYusz+CR2lr6Jk0uhiT8MfPzOcw5J0gE18Z1v5zc2vrG5lZ+u7Czu7dUDw8auo4VQwbLBaxagdUo+ASG4Ybge1EIY0Cga1gdD3NWeoNIlrRkn6Ed0IHnIGTXWqnu9YsktuzORVfAWULr6ePiuAkCtVzs9mOWRigNE1Trjucmxs+oMpwJnBS6qcaEshEdYMeipBFqP5sNOiFn1umTMFb2SUNm7u+OjEZaj6PAVkbUDPVyNjXyzqpCSt+xmWSGpRslGYCmJiMt2a9LlCZsTYAmWK21kJG1JFmbG3KdgjeMsrr0LzouxZrnulqgtz5eEETuEcPLiEKtxADRrAAOERnuHFuXOenFfnbV6acxY9xBHzvsP6C+PLg==ltlatexitgtltlatexit sha1_base64=G80TKlOggVS1bdO2UARKLbBlog=gtAAAB6HicbZBNSwMxEIYnftb6VfXoJVgET2XXix4LXjy2YD+gXUo2nW1js9klyQpl6Sw4kERr4kb4b03YP2vpC4OGdGTLzhqkUxnreN9nY3Nre2S3tlfcPDo+OKyenbZNkmmOLJzLR3ZAZlEJhyworsZtqZHEosRNO7ub1zhNqIxL1YKcpBjEbKREJzqyzmv6gUvVq3kJ0HfwCqlCoMah89YcJz2JUlktmTM3UhvkTFvBJc7KcxgyviEjbDnULEYTZAvFp3RS+cMaZRo95SlCf3RM5iY6Zx6DpjZsdmtTY36v1MhvdBrlQaWZR8eVHUSapTej8ajoUGrmVUweMa+F2pXzMNOPWZVN2IfirJ69D+7rmO2761bpXxFGCc7iAKDhBupwDw1oAQeEZ3iFNJIXsg7+Vi2bpBi5gz+iHz+AHRFjKA=ltlatexitgt

0ltlatexit sha1_base64=y7+k6C9rYTv+3IMOPjyUOmZXQ9k=gtAAAB6HicbZC7SgNBFIbPxluMt6ilzWAQrMKujekM2FgmYC6QLGF2cjYZMzu7zMwKcckT2FgoYusz+CR2lr6Jk0uhiT8MfPzOcw5J0gE18Z1v5zc2vrG5lZ+u7Czu7dUDw8auo4VQwbLBaxagdUo+ASG4Ybge1EIY0Cga1gdD3NWeoNIlrRkn6Ed0IHnIGTXWqru9YsktuzORVfAWULr6ePiuAkCtVzs9mOWRigNE1Trjucmxs+oMpwJnBS6qcaEshEdYMeipBFqP5sNOiFn1umTMFb2SUNm7u+OjEZaj6PAVkbUDPVyNjXyzqpCSt+xmWSGpRslGYCmJiMt2a9LlCZsTYAmWK21kJG1JFmbG3KdgjeMsrr0LzouxZrnulqgtz5eEETuEcPLiEKtxADRrAAOERnuHFuXOenFfnbV6acxY9xBHzvsP5quPLQ==ltlatexitgtltlatexit sha1_base64=kDJXD6ZgKJHcoD6HUKLL1QORk=gtAAAB6HicbZBNSwMxEIZnaz1q+rRS7AInkrWix4LXjy2YD+gXUo2nW1js9klyQpl6Sw4kERr4kb4b03YP2vpC4OGdGTLzhqkUxlL67W1sbm3v7Jb2yvsHh0fHlZPTtkkyzbHFE5nobsgMSqGwZYWV2E01sjiU2AkndN65wm1EYl6sNMUg5iNlIgEZ9ZZTTqoVGmNLkTWwS+gCoUag8pXf5jwLEZluWTG9Hya2iBn2goucVbuZwZTxidshD2HisVognyx6IxcOmdIokS7pyxZuL8nchYbM41D1xkzOzartbn5X62X2eg2yIVKM4uKLz+KMklsQuZXk6HQyK2cOmBcC7cr4WOmGbcum7ILwV89eR3a1zXfcdOv1mkRRwnO4QKuwIcbqMM9NKAFHBCe4RXevEfvxXv3PpatG14xcwZ5H3+AHLBjJ8=ltlatexitgt




-

+

One-hot User Encoding

Encoder

h(1)i1ltlatexit sha1_base64=n8L2xGBp3Se47W3W8ymPZR9ACU=gtAAAEw3icbVPNTtwwEDawbSn9gbWXqxGSKDS1SYc6BGVFlWqtqL8LEhJWDmOw1rrOIvtFFYhL1Kph17a1+hr9G06TpYtWbCUaGb8fTOfx+NoJLg2nc7fufmF1r37DxYfLj16OTps+WV5z2d5YqyI5qJTJ1ERDPBJTsy3Ah2MlKMpJFgx9Fwx+4ff2NK80wemvGIhSk5kzzhlBgI9ZdXgl4xKE+LNXe97Be875b9ZafT7lQL3zbcieFsvmJT8vtb6KwvfgzijecqkoYJo7budkQkLogyngpVLQa7ZiNAhOWM+mJKkTIdFpb3EqxCJcZIp+KTBVfQmoyCp1uM0AmRKzEDP7tngdG8VHxjcJWqM9zMSY8vLhK6ymwHLFANFRECRdJQbprCmnEnKGjm1SSGBiidJm+pN8i4suLRsSWvxSS6wybBtLo65YtSIMRiEKg7VMB0QRSgUA3Ugj53ntiImMgbHLAWSXYCcFPwi0MyUvhsWQeG4QVnObp6Xvhf6MiwKxyv77kYg4szojcoDBhBW8ZdMpU2ehEhF9CDxleMFV1PwrsoiJnmu76AlNc+qsZaG05ZnIl4BmxDVRGABbZtUWThp1ab5exAxwW7fJvc5trLsHXswepEs601j4j4i4SjLiYIe1PSMA6ZFLbKbrJOKzPYrXBCNTQBuDzBCFEBBc2ZAY7Lg6UmnhWTQ8udzZvr2bBnMX1uF1L7xKj+GUT3P0vIkru0LBLKNDK6qq6dQPXHG+9ynfQ+9AEQ6D0N6+R9u9V83yNOASJg7OaV+zOt2bxs9r+1utr2v8Kzfo3otolfoNVpDLtpC2+gT2kNHiKIL9AP9Qr9bH1vDlmqZGjoN+G8QI3VKv8Bp8iX8A==ltlatexitgt

h(2)i2ltlatexit sha1_base64=cRyQTkApPuf6d562OrWLfC8DipI=gtAAAEw3icbVPNbtQwEHbpAqX8tXCDi0VUqYhSbdJDOVYUKiS0qPRnWylJV44z27XWcba2l3aV5kWQOHCB1+A1eBvGyXbpbmsp0Xj8fTOfPTPJQApjm82c3fmG3fv3V94sPjw0eMnT5eWn7VNPtQcDnkuc32cMANSKDi0wko4HmhgWSLhKOlvuOjb6CNyNWBHQ0gztipEl3BmUVXZ2k5ahe98qRYDV6XnUJ0grKz5DXXm9WiNw1bHhbb37S8M+Lzd3O8vz3KM35MANluWTGhH5zYOOCaSu4hHIxGhoYMN5npxCiqVgGJi4q7SVdQU9Ku7nGT1laea8zCpYZM8oSRGbM9szsmXNOzlbovqUtpkd0L2cpdbxcmiq67UGuARUxiUmywdCCpoYLUBymYhqbYQCdjoNOq7fdd3EhlGMrXovvDiW1OXWPS1OhgVs5QoNxLTAb5T2mGcdkqA7lwdnQZaRMpbixi5GCc5ST4b6IDNgy9OMiKjwKsvZw7MyDOJQxUXhYZn8tUimuTVr1Q4ZSFihX3KdTfMUeipigIEvvSC6nIB3dJ6AEkNzC61b85waZ4U78YQlQKYzYOeqkiAscs+WJA5+4rQ5zja+uISLt92bXFcMl8ddrA4Ubk9y7QGTt5GwxeUMaW9MQtYBKOO66DrjoL6L04YtUEOnAJHCCkTLFgfLPV8Gmk93jk1bSzubNx2zcI+S+t2u5LeYlaLi2lw67+IpHuLhh3GkVZWpWrVD7jq4WS6ePvtD9NgdJThxhXSYPqv1GeREJhx+E93TT7s7N702gH67GevAVxo9qdcCeUlekVXik02yRT6RXXJIODknP8gv8rvxsdFv6IatoXfmxpznZGo1yn+wa5fyltlatexitgt

Figure 2 Overall network architecture All convolutional networks share the same weights For each type of items a type-dependent

encoder is used to learn binary codes The matching block is responsible for computing the final preference the user has for the outfit

where i = (i1 iN ) is the index tuple

The fashion taste can be very different for different users

ie the fashion style preference is personal (see Fig 1)

The personalized fashion preference of a certain user can

be implicitly represented by outfits the user created or rated

in the past We use ruOito indicate the preference of user

u to outfit Oi The higher the score the more the user likes

that outfit Our task is to predict ruOifor any user and outfit

pair so that the most preferable outfits can be recommended

to users

Without loss of generality letrsquos assume that an outfit is

composed of N items from N categories Then the total

number of possible user-outfit pairs is U times L1 times middot middot middottimes LN

which is an extremely large number in practice There-

fore we need to compute ruOiin a more feasible way so

that some efficient search technique can be applied In this

work we explore the binary embedding technique ie we

seek to represent users and fashion items with binary codes

from which the preference scores are computed

32 Overall framework

We propose a fashion hashing network (FHN) to predict

the preference of users to different outfit compositions The

overall architecture is shown in Fig 2 It consists of three

types of components a feature network for feature extrac-

tion several type-dependent hashing modules to learn bi-

nary codes and a matching block to predict the preference

score For input each user is represented by a one-hot vec-

tor indicating the index of the user We use a convolutional

network to extract visual features for images The feature

networks for different item categories share the same pa-

rameters The specific structure of this feature network is

optional to us Note that textual information can also be in-

corporated which we will discuss later Items from different

categories and the users are considered as different types

Each hashing module consists of some fully-connected lay-

ers with a sign function for binarization The last compo-

nent is a matching block to compute the preference score

given the binary codes There are two terms contributing to

the final score One only considers item compatibilities and

the other takes usersrsquo taste into consideration The detailed

formulation is discussed in the following subsections

33 The matching block

The compatibility among the items of an outfit and the

fitness to a user are multiway relationships Although many

efforts have been made to the model high order relation-

ships in various applications the most successful way to

handle them is still to decompose it into pairwise relation-

ships which is easier to learn and has been proven to be a

good approximation Considering the efficiency of comput-

ing the preference scores and retrieving compatible outfits

for recommendation we also build our model based on pair-

wise interaction relations

Let bi bj isin minus1+1D be the binary codes of two ob-

jects which can be a fashion item or a user their compati-

bility is measured by

mij = b⊺

i Λbj (2)

where Λ is a weighting matrix which we constrain it to be

diagonal ie Λ = diag(λ1 λD) Λ is introduced to

better capture the relationship between objects while main-

taining the computational efficiency of binary codes And

the score for outfit Oi with respect to user u is computed by

10564

ruOi= α middot r

(u)uOi

+ r(i)uOi

(3)

where

r(u)uOi

=1

z1

983131

n

b(n)⊺in

Λ(u)b(u)u (4)

r(i)uOi

=1

z2

983131

n

983131

m

b(n)⊺in

Λ(i)b

(m)im

(5)

In the above formulations Λ(u) and Λ(i) are the weight-

ing matrices for user-item and item-item pairs respectively

The Eq (4) models userrsquos preference to the fashion items

and Eq (5) measures the compatibility between pairs of

fashion items They are normalized by the number of pairs

involved ie z1 and z2 The scalar α is used to balance the

contributions of the two terms Although α can be absorbed

into Λ(u) we keep it in Eq (3) for later discussion of not

using weighted hashing ie Λ(u)Λ(i) are set to identity

matrix I

34 Learn to hash

Directly optimizing an objective function with the binary

constraint on b(n)in

b(u)u is challenging It is common to re-

lax the problem by using continuous variables to replace

them during optimization The binary codes are obtained

by taking the sign of the continuous variables ie

b(n)in

= sign(h(n)in

) b(u)u = sign(h(u)u ) (6)

where sign(x) = 1 if x ge 0 and minus1 otherwise

Many hashing methods learn the hash code by mini-

mizing the quantization error after binarization Recently

Cao et al [1] propose a HashNet method which approx-

imates the sign operation through activation in the neu-

ral network This method avoids explicitly tackling the

quantization error Following this work we rewrite Eq (4)

and Eq (5) as follows

r(u)uOi

=1

z1

983131

n

b(n)⊺

intΛ

(u)b(u)

ut (7)

r(i)uOi

=1

z2

983131

n

983131

m

b(n)⊺

intΛ

(i)b(m)

imt (8)

where

b(n)

int= tanh(βth

(n)in

) b(u)

ut = tanh(βth(u)u ) (9)

and βt is a scalar that increases with the iteration t during

optimization When βt is large Eq (9) is a close approxi-

mation of Eq (6) (see the diagram in Fig 2) So b(n)

intand

b(u)

ut converge to the binary codes b(n)in

and b(u)u when the

optimization ends

35 Objective function

Besides images the fashion items are usually also de-

picted by some textual descriptions when exhibited online

These textual descriptions provide semantic information for

the images which would be very helpful for compatibility

modeling We therefore also use features extracted from

textual content in out model

We use the same way to convert textual features into

binary codes and compute the corresponding preference

scores Suppose binary codes from different modalities

are donated by b(n)vin

b(n)fin

where v and f indicate vi-

sual and textual respectively The overall score with multi-

modality information is computed by adding the scores of

each modality

ruOi(b

(n)vin

b(u)u ) + ruOi(b

(n)fin

b(u)u ) (10)

The scores of each modality is computed by Eq (3) and the

same binary codes of users are used for different modalities

We use ranking loss to learn the parameters of our model

ie the network is trained so that given a pair of outfits it

can predict which one is preferable to a user The training

set contains a set of outfit pairs

P equiv (u i j)|ruOigt ruOj

(11)

where (u i j) indicates that user u prefers outfit Oi over

Oj We adapt the BPR [23] optimization criterion to max-

imize the posterior probability of model parameters The

objective can be expressed as

ℓBPR =983131

(uij)isinP

log983043

1 + exp(minus(ruOiminus ruOj

))983044

(12)

Following previous work [3] we also add constraints

to make the embeddings of visual and textual information

more consistent to each other For simplicity suppose vf

are binary codes for items from the two modalities Their

similarity is donated by s(vf) = v⊺f We require that

v and f that correspond to the same item should be more

similar than those for different items This is achieved by

minimizing the following loss

ℓV SE =983131

vk

max0 cminus s(vf) + s(vfk)+ (13)

983131

f k

max0 cminus s(vf) + s(vkf)

where (vf) are for the same item (vfk) are for different

items and so is (vkf)Overall the objective of the proposed method is

minΘ

E(ℓBPR + λℓV SE) (14)

10565

Polyvore-630 Polyvore-53

Splits Outfits Items Outfits Items

Train 127326 159729 10712 20230

Test 23054 45505 1944 4437


Splits Outfits Items Outfit Items

Train 83416 146475 5133 14594

Test 14654 39085 898 2797

Table 1 Statistics of our Polyvore datasets

where Θ are parameters for the network and λ is a weighting

parameter Θ consists of ΘnnΘvΘf Θu where Θnn

are parameters of the feature network and ΘvΘf Θu are

parameters of the encoders for the two modalities and for

the users respectively The optimization problem is solved

with continuous relaxation as discussed in Sec 34

36 Implementation details

The structure of the feature network is optional for us

In our experiments we simply use AlexNet [9] as the back-

bone To handle images with arbitrarily sizes we replace all

fully-connected layers in AlexNet with convolutional layers

and an average pooling layer is added to get a fixed feature

dimension of 4096 The dimension for textual feature is

2400 The type-dependent hashing modules for items con-

sist of two fully-connected layers The encoder for users

contains one fully-connected layer Our methods are imple-

mented in PyTorch [22]

4 Polyvore Dataset

41 Polyvore-U

Since existing datasets [7 3 27] are either too small or

lacking the user information they cannot be used for our

personalized fashion outfit recommendation problem We

collect a new dataset from the Polyvore website We let

each outfit contain items from three categories ie top

bottom and shoes We created 4 versions of the datasets

denoted by Polyvore-U where U is the number of users

Two datasets ie Polvvore-630 and Polyvore-53 contain

outfits with a fixed number of items ie each outfit has

one and only one item from each category In the other

two datasets Polvvore-519 and Polyvore-32 the number

of items in each outfit is varying ie some outfits may

have two tops The two larger datasets Polvvore-630 and

Polvvore-519 are used for most experiments Polyvore-53

and Polyvore-32 are reserved to test the user generalization

ability of our models The statistic of our data sets is shown

in Table 1

42 Data preparation

We take outfits created by a user as positive outfits for

that user The negative outfits for himher come from two

sources One is random mixtures of items and the other is

random samples of other usersrsquo positive outfits The sec-

ond type of negative outfits are more difficult than the first

one Outfit composition methods that do not consider the

personalization issue usually fail to distinguish them from

positive outfits For fair comparison we first only include

the easy negative outfits when comparing the performance

of different methods We discuss the results with the hard

negative outfits separately in Sec 53 The ratio between

negative and positive outfits is set to 101 for each user The

number of items in a negative outfit is kept consistent to that

in a positive outfit in each dataset We ensure that this is no

overlap of items between the training and testing sets for

each user

5 Experiment Results

51 Evaluation metric

We conduct experiments on two recommendation tasks

The first is outfit recommendation ie for each user we rank

the testing outfits in descending order of their compatibility

scores The ranking performance is evaluated by Area Un-

der the ROC curve (AUC) and Normalized Discounted Cu-

mulative Gain (NDCG) The second is the fill-in-the-blank

(FITB) fashion recommendation experiment The goal is to

select an item from a set of candidate items (four in our ex-

periments) that is most compatible with the remaining items

of the outfit The ground truth item is the correct answer and

the performance is measured by accuracy of the answers

For each experiment we report the average results over all

users

52 Performance comparison

We compare variant versions of our models with three

state-of-the-art methods

bull SiameseNet [28] utilizes a Siamese CNN to learn a

feature transformation to the latent style space which

maintains the matching relationship for pairs of items

The score of an outfit is obtained by averaging the pair-

wise similarities The embedding size is set to 512

bull Bi-LSTM [3] uses bidirectional LSTM to learn

the compatibility of an outfit by considering the

items as a sequence Image features are extracted

from Inception-V3 [26] and transformed into represen-

tation with 512 dimension before fed into LSTM

bull CSN [27] maps pairs of items into type-specific em-

bedding spaces Compatibility is measured by dis-

tances in these spaces An extra distance metric ie

10566

(a) User 1

(b) User 2

(c) User 3

Figure 3 Top-10 outfits with the highest scores computed by different methods on Polyvore-630 The outfits in red boxes are positive

outfits and those in black boxes are negative ones

a weighted inner product is also learned to replace the

Euclidean distance We use 512 for embedding size

and their backbone is ResNet-18 [4]

bull FHN is our fashion hashing net We use AlexNet as

backbone and use weighted hashing to compute the

compatibility between items and users The length of

binary code D is set to 128 for all schemes We evalu-

ate four different types of weighting schemes

- FHN-T0 set Λ(u) = Λ(i) = I

- FHN-T1 set Λ(i) = Iα = 1

- FHN-T2 set Λ(u) = Iα = 1

- FHN-T3 set α = 1

We use the two larger datasets Polyvore-630 and

Polyvore-519 for the comparison of different methods The

results are shown in Table 2 Here we also evaluate the

contribution of each modality in our model The textual

features are obtained from the descriptions and tags of the

items using seq2seq [25] To show the contribution of each

modality we train FHN-T3 with each modality separately

and show the results in part (b) And all methods in part

(c) employ both visual and textual information From the

results we can see that all our full-version methods outper-

form the state-of-the-arts methods under all metrics Even

with only vision features our model still works better than

other methods By comparing results of the two modalities

we find that the visual information is more helpful than tex-

10567


Methods FITB AUC NDCG FITB AUC NDCG

(a) SiameseNet [28] 05103 07703 06109 05304 08026 06648

Bi-LSTM [3] 05515 08102 06629 05232 07746 06210

CSN [27] 05536 08187 06744 05617 08215 06703

(b) FHN-T3 (Textual) 05144 08441 07343 04857 08188 06953

FHN-T3 (Visual) 06052 08942 08090 06035 08845 07784

(c) FHN-T0 06066 08989 08213 05770 08761 07821

FHN-T1 06159 09016 08251 06062 08892 08014

FHN-T2 06451 09027 08296 06283 08975 08184

FHN-T3 06461 09176 08541 06386 09137 08448

Table 2 Comparison of different methods on Polyvore-630 and Polyvore-519

Dataset SiameseNe Bi-LSTM CSN

Polyvore-630 9778 9460 9476

Polyvore-519 9711 9750 9615

Table 3 Wining rate () of FHN-T3 over other methods

tual in this task And combining the two modalities leads to

better performance than only using one of them

Overall the proposed methods with multi-modality get

66 sim 121 improvement in AUC and 1956 sim

2665 improvement in NDCG when compared with the

best results of other methods on the two datasets To visual-

ize the ranking quality we show top-10 outfits of three users

with the highest scores in Fig 3 FHN usually has better

ranking results To show how good FHN is when compared

to other methods we define the winning rate as the per-

centage of users one method outperforms the other in mean

NDCG We show the comparisons in Table 3 It shows that

FHN has better ranking results for at least 94 users The

improvement mainly comes from the personalized model-

ing of usersrsquo fashion preferences It is also beneficial to use

the BPR optimization criterion which explicitly takes rank-

ing into consideration FHN-T0 uses unweighted hashing

in Eq (3) which leads to a relatively poor performance as

shown in Table 2 And as expected adding weighting to

hashing improves the results Without loss of generality in

the following we only consider FHN-T3 and make it the

default setting for analysis

53 Performance on hard outfits

As mentioned in Sec 42 there are two types of nega-

tive outfits A challenging case is to use the outfits that are

posted by other users as negative outfits for the current user

in evaluation This setting is different from that of previ-

ous work where all user created outfits are taken as posi-

tive ones We learn usersrsquo preferences through the first term

in Eq (3) Note that only 128 extra bits are introduced to

characterize the users The results are shown in Table 4

Polyvore-630-H Polyvore-519-H

Methods AUC NDCG AUC NDCG

SiameseNet 04993 02808 04997 02731

Bi-LSTM 04992 02817 04990 02739

CSN 05000 02790 04995 02740

FHN 07654 05552 07550 05369

FHN-H 08440 06869 08361 06685

Table 4 Results on hard negative outfits FHN-H utilizes hard

negative outfits during training while FHN does not include hard

negative outfits during training

FHN indicates results obtained without including hard neg-

ative outfits during training And FHN-H involves hard neg-

ative outfits in the training set We can see that the baseline

methods perform poorly in this experiment since they re-

gard all hard negatives as positive ones Our method works

much better with the same training set By including hard

negatives during training the performance can be further

improved

54 Learning hashing codes for cold-start users

Code start is a common problem in recommendation sys-

tems New users joins constantly in a social network It will

be un-affordable to retrain the whole network for each new

user To tackle this problem we keep the feature network

for items fixed and only retrain the user representations for

newcomers which is only a 128 bits binary code in our

method and can be computed very efficiently We evaluate

a scenario where the system have built a model for 630529


FITB 05998 05780

AUC 08890 08911

NDCG 08211 07970

Table 5 Results of learning hash codes for new users

10568



FHN w Eq (5) 05530 08180 06637 04836 08289 06949

FHN d Eq (4) 05733 08333 07037 05747 08334 07006

FHN w Eq (4) 05062 08463 07274 05578 08243 06717

FHN d Eq (5) 05302 08571 07609 05170 08519 07449

FHN-T3 in Table 2 06461 09176 08541 06386 09137 08448

Table 6 The contribution of each term in Eq (3) FHN w Eq (5) is trained only using Eq (5) and FHN w Eq (4) is trained only

using Eq (4) FHN d Eq (4) and FHN d Eq (5) drop the corresponding term of a trained FHN model

Figure 4 Comparison of different lengths of codes on Polyvore

datasets

users in the past and 5332 new users come The results

are reported in Table 5 Compared with the performance

on Polyvoe-630519 the performance drops are acceptable

ie our method can maintain the performance by only fine-

tuning the new usersrsquo representations even when the size of

the dataset has grown around 7

55 Performance with different lengths of codes

A hashing code with D bits can distinguish at most 2D

objects Usually with the increase of code length the per-

formance will be promoted Here we illustrate the influ-

ence of code length on our approach We evaluate a wide

range of code lengths ie 16 32 64 128 256 and show

their performance comparison in Fig 4 Taking the AUC for

example the improvement is roughly proportional to the log

of the code length Too short code gets poor result For ex-

ample when D = 16 the maximum number of items it can

represent is 216 = 65 536 which is smaller than the num-

ber of the items in the datasets we use Thus the accuracy

drops significantly with D = 16

56 Ablation analysis

As presented in Eq (3) we argue that the ranking score

consists of both item-item and user-item compatibilities To

evaluate the contribution of each term we do ablation study

by training with only one term If we only use Eq (5) it de-

generates to an unpersonalized outfit composition method

And if only using Eq (4) it only captures userrsquos preference

to individual items and the compatibility between items

would not be modeled Besides we also evaluate the per-

formance after dropping one term after a full FHN model

is trained This is to make sure that no term overtakes the

other in FHN The results are shown in Table 6 We can

see that all results are worse than the full FHN model This

demonstrates that every term is indispensable in the model

6 Conclusion

In this paper we study how to utilize the hashing tech-

nique for efficient personalized fashion outfit recommenda-

tion Although there are numerous ways to represent the

compatibility of outfits this problem needs to be well han-

dled to fit into hashing optimization We propose a for-

mulation based on weighted pairwise relations We de-

sign category-dependent hashing mapping for items and

users and train the whole framework in an end-to-end man-

ner Meanwhile we use a simple way to combine multi-

modality information to improve the performance Through

extensive experiments on a large scale Polyvore dataset we

show the superiority of the proposed method over the state-

of-the-art methods even with a simple backbone and binary

representation

10569

References

[1] Zhangjie Cao Mingsheng Long Jianmin Wang and Philip S

Yu HashNet Deep Learning to Hash by Continuation In

ICCV 2017

[2] Jian Dong Qiang Chen Xiaohui Shen Jianchao Yang and

Shuicheng Yan Towards Unified Human Parsing and Pose

Estimation In CVPR 2014

[3] Xintong Han Zuxuan Wu Yu-Gang Jiang and Larry S

Davis Learning Fashion Compatibility with Bidirectional

LSTMs In ACM MM 2017

[4] Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun

Deep Residual Learning for Image Recognition In CVPR

2016

[5] Wei-Lin Hsiao and Kristen Grauman Learning the Latent

rdquoLookrdquo Unsupervised Discovery of a Style-Coherent Em-

bedding from Fashion Images In ICCV 2017

[6] Wei-Lin Hsiao and Kristen Grauman Creating Capsule

Wardrobes from Fashion Images In CVPR 2018

[7] Yang Hu Xi Yi and Larry S Davis Collaborative Fashion

Recommendation A Functional Tensor Factorization Ap-

proach In ACM MM 2015

[8] Vignesh Jagadeesh Robinson Piramuthu Anurag Bhardwaj

Wei Di and Neel Sundaresan Large Scale Visual Recom-

mendations From Street Fashion Images In KDD 2014

[9] Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton Im-

ageNet Classification with Deep Convolutional Neural Net-

works In NeurIPS 2012

[10] Hanbit Lee Jinseok Seol and Sang-goo Lee Style2Vec

Representation Learning for Fashion Items From Style Sets

arXiv 2017

[11] Wu-Jun Li Sheng Wang and Wang-Cheng Kang Feature

Learning Based Deep Supervised Hashing with Pairwise La-

bels In IJCAI 2016

[12] Yuncheng Li Liangliang Cao Jiang Zhu and Jiebo Luo

Mining Fashion Outfit Composition Using an End-to-End

Deep Learning Approach on Set Data TMM 2017

[13] Defu Lian Rui Liu Yong Ge Kai Zheng Xing Xie and

Longbing Cao Discrete Content-aware Matrix Factoriza-

tion In KDD 2017

[14] Xiaodan Liang Liang Lin Wei Yang Ping Luo Jun-

shi Huang and Shuicheng Yan Clothes Co-Parsing Via

Joint Image Segmentation and Labeling With Application to

Clothing Retrieval TMM 2016

[15] Haomiao Liu Ruiping Wang Shiguang Shan and Xilin

Chen Deep Supervised Hashing for Fast Image Retrieval

In CVPR 2016

[16] Si Liu Jiashi Feng Zheng Song Tianzhu Zhang Hanqing

Lu Changsheng Xu and Shuicheng Yan Hi Magic Closet

Tell Me What to Wear In ACM MM 2012

[17] Si Liu Luoqi Liu and Shuicheng Yan Fashion Analysis

Current Techniques and Future Directions IEEE MultiMe-

dia 2014

[18] Si Liu Zheng Song Guangcan Liu Changsheng Xu Han-

qing Lu and Shuicheng Yan Street-to-Shop Cross-Scenario

Clothing Retrieval via Parts Alignment and Auxiliary Set In

CVPR 2012

[19] Xianglong Liu Junfeng He Cheng Deng and Bo Lang Col-

laborative Hashing In CVPR 2014

[20] Ziwei Liu Ping Luo Shi Qiu Xiaogang Wang and Xiaoou

Tang DeepFashion Powering Robust Clothes Recognition

and Retrieval with Rich Annotations In CVPR 2016

[21] Julian J McAuley Christopher Targett Qinfeng Shi and An-

ton van den Hengel Image-Based Recommendations on

Styles and Substitutes In SIGIR 2015

[22] Adam Paszke Sam Gross Soumith Chintala Gregory

Chanan Edward Yang Zachary DeVito Zeming Lin Al-

ban Desmaison Luca Antiga and Adam Lerer Automatic

differentiation in PyTorch 2017

[23] Steffen Rendle Christoph Freudenthaler Zeno Gantner and

Lars Schmidt-Thieme BPR Bayesian Personalized Ranking

from Implicit Feedback In UAI 2009

[24] Edgar Simo-Serra and Hiroshi Ishikawa Fashion Style in

128 Floats Joint Ranking and Classification Using Weak

Data for Feature Extraction In CVPR 2016

[25] Ilya Sutskever Oriol Vinyals and Quoc V Le Sequence

to Sequence Learning with Neural Networks In NeurIPS

2014

[26] Christian Szegedy Vincent Vanhoucke Sergey Ioffe

Jonathon Shlens and Zbigniew Wojna Rethinking the In-

ception Architecture for Computer Vision In CVPR 2016

[27] Mariya I Vasileva Bryan A Plummer Krishna Dusad

Shreya Rajpal Ranjitha Kumar and David A Forsyth Learn-

ing Type-Aware Embeddings for Fashion Compatibility In

ECCV 2018

[28] Andreas Veit Balazs Kovacs Sean Bell Julian McAuley

Kavita Bala and Serge Belongie Learning Visual Clothing

Style with Heterogeneous Dyadic Co-Occurrences In ICCV

2015

[29] Jingdong Wang Ting Zhang Jingkuan Song Nicu Sebe and

Heng Tao Shen A Survey on Learning to Hash TPAMI

2017

[30] Qifan Wang Dan Zhang and Luo Si Weighted Hashing for

Fast Large Scale Similarity Search In CIKM 2013

[31] Kota Yamaguchi M Hadi Kiapour Luis E Ortiz and

Tamara L Berg Retrieving Similar Styles to Parse Clothing

TPAMI 2015

[32] Hanwang Zhang Fumin Shen Wei Liu Xiangnan He

Huanbo Luan and Tat-Seng Chua Discrete Collaborative

Filtering In SIGIR 2016

[33] Jian Zhang and Yuxin Peng Query-Adaptive Image Re-

trieval by Deep-Weighted Hashing TMM 2018

[34] Lei Zhang Yongdong Zhang Jinhu Tang Ke Lu and Qi

Tian Binary Code Ranking with Weighted Hamming Dis-

tance In CVPR 2013

[35] Ke Zhou and Hongyuan Zha Learning Binary Codes for

Collaborative Filtering In KDD 2012

[36] Han Zhu Mingsheng Long Jianmin Wang and Yue Cao

Deep Hashing Network for Efficient Similarity Retrieval In

AAAI 2016

10570

comes an extremely important problem The items and out-

fits should be stored in an economical way The compati-

bility of an outfit should be evaluated not only accurately

but also efficiently And the most compatible items and

outfits need to be retrieved swiftly These are issues that

havenrsquot been well handled by previous works In this work

we take them into consideration and explore the hash tech-

nique for fashion recommendation Learning to hash has

been extensively studies for efficient image retrieval [15]

Some recent works have also combine it with collabora-

tive filtering algorithms to recommend individual items to

users [35 19 32 13] We incorporate it into the task of sets

composition problem for personalized outfit recommenda-

tion

We design a neural network for efficient personalized

fashion outfit recommendation The network captures the

favors of different users and learn the compatibility between

fashion items It is composed of three components A fea-

ture network is first used to extract content features Then a

set of type-dependent hashing modules convert the features

and user taste representations into binary codes Finally

the overall preference of a user to an outfit is computed by

a matching block that conducts pairwise weighted hashing

matching Both visual and textual information are utilized

in our model Since existing datasets are either two small

or lacking user information we collect a new dataset which

contains more than 138000 outfits created by hundreds of

online users We evaluate our method on this large scale

dataset Extensive experiments show that it outperforms the

state-of-the-art methods even with a simple backbone and

binary representations

2 Related Work

21 Fashion recommendation

Fashion analysis [17] such as clothing recognition [20]

latent embedding [24 5 10] parsing [2 31 14] re-

trieval [18 14] and recommendation [16 8 21 28 6] have

attracted many attentions in recent years Among the plenty

of works that studied fashion related problems we focus on

those that related to the recommendation task in the follow-

ing

Liu et al [16] introduce a latent SVM based model for

occasion-oriented clothing recommendation McAuley el

al [21] propose a parametric distance transformation to

learn the item compatibility Veit et al [28] utilize Siamese

network to learn embedding for fashion items These works

do not consider the composition of multiple items in an out-

fit and only focus on the matching between two items Some

methods seek to directly model the high-order relationships

between items Li et al [12] apply multi-modality fusion

and RNN-based multi-instance pooling models to classify

the outfit quality Han et al train a bidirectional LSTM as

a scorer to predict the compatibility of an outfit Vasileva et

al [27] learn type-aware mapping for different kinds of

items and utilize metric layers to learn the compatibility

As for personalized composition problem Hu et al [7]

make an initial exploration A functional tensor factoriza-

tion method is proposed to model the user-item and item-

item interactions in multiple latent spaces Nevertheless

they use hand-crafted features and do not jointly optimize

the representation of images Hsiao el al [6] propose a sub-

set selection model for selecting a minimal set of garments

that maximize the compatibility and versatility which in-

troduce a new recommendation topic However those men-

tioned models do not consider the efficiency of the system

22 Learning to hash

Learning to hash is the task of learning a compact bi-

nary code for the input item It aims to maintain the near-

est neighbor relation of the original space in the hamming

space Hashing methods have become a promising and pop-

ular technique for efficient similarity search which also re-

duce the storage cost of data The basic idea in learning to

hash is similarity preserving [29] ie minimizing the gap

between the similarity computed in hash-coded space and

the similarity in the original space Most existing hashing

methods first introduce some relaxations to their problems

by learning real-valued embedding and then take the sign of

the values to obtain binary codes [15 36 11] which how-

ever often suffer from quantization loss Recently Cao et

al [1] proposed a method to learn hash codes by continua-

tion with convergence guarantees

There have been some works reported that apply hashing

technique to the recommendation problem [35 13] Zhou et

al [35] learn binary code that preserves the preference of

users to items in collaborative filtering Lian et al [13] pro-

pose a discrete content-aware matrix factorization model

However these methods cannot be applied directly to the

outfit composition problem since they only recommend sin-

gle items while an outfit contains multiple interacted items

In this work we model outfit compatibility through pair-

wise interactions and employ the weighted hashing tech-

nique [34 30 33] for matching users and items

3 The Proposed Approach

31 Problem formulation

Suppose there are N fashion categories (eg top bot-

tom and shoes) The number of items in the n-th cate-

gory is donated by Ln and the number of users is U Let

X(n) = x(n)1 x

(n)Ln

donate all items in the n-th cate-

gory where x(n)i is the i-th item in it Then an outfit with N

items with each from one category can be represented as

Oi = x(1)i1

x(2)i2

x(N)iN

(1)

10563

Fashion Outfit

Visual

Embedding

Type-dependent

Encoder

Binary Codes

Shared Parameters

User

Embedding

Sign Activation

Approximated Sign

Λ(i)


Λ(i)


Λ(u)


Λ(u)


Matching Block



EncoderFeature

CNN+1













-

+

b(3)i3


EncoderFeature

CNNminus1













-

+

b(2)i2


EncoderFeature

CNNminus1













-

+

b(1)i1
































-

+


Encoder














to users














































mij = b⊺

i Λbj (2)






10564

ruOi= α middot r

(u)uOi

+ r(i)uOi

(3)

where

r(u)uOi

=1

z1

983131

n

b(n)⊺in

Λ(u)b(u)u (4)

r(i)uOi

=1

z2

983131

n

983131

m

b(n)⊺in

Λ(i)b

(m)im

(5)










matrix I

34 Learn to hash







b(n)in

= sign(h(n)in

) b(u)u = sign(h(u)u ) (6)









r(u)uOi

=1

z1

983131

n

b(n)⊺

intΛ

(u)b(u)

ut (7)

r(i)uOi

=1

z2

983131

n

983131

m

b(n)⊺

intΛ

(i)b(m)

imt (8)

where

b(n)

int= tanh(βth

(n)in

) b(u)





intand

b(u)


and b(u)u when the

optimization ends












b(n)fin




each modality

ruOi(b

(n)vin

b(u)u ) + ruOi(b

(n)fin

b(u)u ) (10)








(11)





ℓBPR =983131

(uij)isinP

log983043


))983044

(12)









ℓV SE =983131

vk


983131

f k




minΘ


10565



Train 127326 159729 10712 20230

Test 23054 45505 1944 4437



Train 83416 146475 5133 14594

Test 14654 39085 898 2797



















4 Polyvore Dataset

41 Polyvore-U

















in Table 1

42 Data preparation
















each user















users

















10566

(a) User 1

(b) User 2

(c) User 3














- FHN-T3 set α = 1















10567



(a) SiameseNet [28] 05103 07703 06109 05304 08026 06648

Bi-LSTM [3] 05515 08102 06629 05232 07746 06210

CSN [27] 05536 08187 06744 05617 08215 06703

(b) FHN-T3 (Textual) 05144 08441 07343 04857 08188 06953

FHN-T3 (Visual) 06052 08942 08090 06035 08845 07784

(c) FHN-T0 06066 08989 08213 05770 08761 07821

FHN-T1 06159 09016 08251 06062 08892 08014

FHN-T2 06451 09027 08296 06283 08975 08184

FHN-T3 06461 09176 08541 06386 09137 08448



Polyvore-630 9778 9460 9476

Polyvore-519 9711 9750 9615



































SiameseNet 04993 02808 04997 02731

Bi-LSTM 04992 02817 04990 02739

CSN 05000 02790 04995 02740

FHN 07654 05552 07550 05369

FHN-H 08440 06869 08361 06685











improved











FITB 05998 05780

AUC 08890 08911

NDCG 08211 07970


10568



FHN w Eq (5) 05530 08180 06637 04836 08289 06949

FHN d Eq (4) 05733 08333 07037 05747 08334 07006

FHN w Eq (4) 05062 08463 07274 05578 08243 06717

FHN d Eq (5) 05302 08571 07609 05170 08519 07449

FHN-T3 in Table 2 06461 09176 08541 06386 09137 08448




datasets


































6 Conclusion














representation

10569

References



ICCV 2017









2016

















arXiv 2017



bels In IJCAI 2016






tion In KDD 2017







In CVPR 2016






dia 2014




CVPR 2012





















2014







ECCV 2018




2015



2017





TPAMI 2015








tance In CVPR 2013





AAAI 2016

10570

Fashion Outfit

Visual

Embedding

Type-dependent

Encoder

Binary Codes

Shared Parameters

User

Embedding

Sign Activation

Approximated Sign

Λ(i)


Λ(i)


Λ(u)


Λ(u)


Matching Block



EncoderFeature

CNN+1













-

+

b(3)i3


EncoderFeature

CNNminus1













-

+

b(2)i2


EncoderFeature

CNNminus1













-

+

b(1)i1
































-

+


Encoder














to users














































mij = b⊺

i Λbj (2)






10564

ruOi= α middot r

(u)uOi

+ r(i)uOi

(3)

where

r(u)uOi

=1

z1

983131

n

b(n)⊺in

Λ(u)b(u)u (4)

r(i)uOi

=1

z2

983131

n

983131

m

b(n)⊺in

Λ(i)b

(m)im

(5)










matrix I

34 Learn to hash







b(n)in

= sign(h(n)in

) b(u)u = sign(h(u)u ) (6)









r(u)uOi

=1

z1

983131

n

b(n)⊺

intΛ

(u)b(u)

ut (7)

r(i)uOi

=1

z2

983131

n

983131

m

b(n)⊺

intΛ

(i)b(m)

imt (8)

where

b(n)

int= tanh(βth

(n)in

) b(u)





intand

b(u)


and b(u)u when the

optimization ends












b(n)fin




each modality

ruOi(b

(n)vin

b(u)u ) + ruOi(b

(n)fin

b(u)u ) (10)








(11)





ℓBPR =983131

(uij)isinP

log983043


))983044

(12)









ℓV SE =983131

vk


983131

f k




minΘ


10565



Train 127326 159729 10712 20230

Test 23054 45505 1944 4437



Train 83416 146475 5133 14594

Test 14654 39085 898 2797



















4 Polyvore Dataset

41 Polyvore-U

















in Table 1

42 Data preparation
















each user















users

















10566

(a) User 1

(b) User 2

(c) User 3














- FHN-T3 set α = 1















10567



(a) SiameseNet [28] 05103 07703 06109 05304 08026 06648

Bi-LSTM [3] 05515 08102 06629 05232 07746 06210

CSN [27] 05536 08187 06744 05617 08215 06703

(b) FHN-T3 (Textual) 05144 08441 07343 04857 08188 06953

FHN-T3 (Visual) 06052 08942 08090 06035 08845 07784

(c) FHN-T0 06066 08989 08213 05770 08761 07821

FHN-T1 06159 09016 08251 06062 08892 08014

FHN-T2 06451 09027 08296 06283 08975 08184

FHN-T3 06461 09176 08541 06386 09137 08448



Polyvore-630 9778 9460 9476

Polyvore-519 9711 9750 9615



































SiameseNet 04993 02808 04997 02731

Bi-LSTM 04992 02817 04990 02739

CSN 05000 02790 04995 02740

FHN 07654 05552 07550 05369

FHN-H 08440 06869 08361 06685











improved











FITB 05998 05780

AUC 08890 08911

NDCG 08211 07970


10568



FHN w Eq (5) 05530 08180 06637 04836 08289 06949

FHN d Eq (4) 05733 08333 07037 05747 08334 07006

FHN w Eq (4) 05062 08463 07274 05578 08243 06717

FHN d Eq (5) 05302 08571 07609 05170 08519 07449

FHN-T3 in Table 2 06461 09176 08541 06386 09137 08448




datasets


































6 Conclusion














representation

10569

References



ICCV 2017









2016

















arXiv 2017



bels In IJCAI 2016






tion In KDD 2017







In CVPR 2016






dia 2014




CVPR 2012





















2014







ECCV 2018




2015



2017





TPAMI 2015








tance In CVPR 2013





AAAI 2016

10570

ruOi= α middot r

(u)uOi

+ r(i)uOi

(3)

where

r(u)uOi

=1

z1

983131

n

b(n)⊺in

Λ(u)b(u)u (4)

r(i)uOi

=1

z2

983131

n

983131

m

b(n)⊺in

Λ(i)b

(m)im

(5)










matrix I

34 Learn to hash







b(n)in

= sign(h(n)in

) b(u)u = sign(h(u)u ) (6)









r(u)uOi

=1

z1

983131

n

b(n)⊺

intΛ

(u)b(u)

ut (7)

r(i)uOi

=1

z2

983131

n

983131

m

b(n)⊺

intΛ

(i)b(m)

imt (8)

where

b(n)

int= tanh(βth

(n)in

) b(u)





intand

b(u)


and b(u)u when the

optimization ends












b(n)fin




each modality

ruOi(b

(n)vin

b(u)u ) + ruOi(b

(n)fin

b(u)u ) (10)








(11)





ℓBPR =983131

(uij)isinP

log983043


))983044

(12)









ℓV SE =983131

vk


983131

f k




minΘ


10565



Train 127326 159729 10712 20230

Test 23054 45505 1944 4437



Train 83416 146475 5133 14594

Test 14654 39085 898 2797



















4 Polyvore Dataset

41 Polyvore-U

















in Table 1

42 Data preparation
















each user















users

















10566

(a) User 1

(b) User 2

(c) User 3














- FHN-T3 set α = 1















10567



(a) SiameseNet [28] 05103 07703 06109 05304 08026 06648

Bi-LSTM [3] 05515 08102 06629 05232 07746 06210

CSN [27] 05536 08187 06744 05617 08215 06703

(b) FHN-T3 (Textual) 05144 08441 07343 04857 08188 06953

FHN-T3 (Visual) 06052 08942 08090 06035 08845 07784

(c) FHN-T0 06066 08989 08213 05770 08761 07821

FHN-T1 06159 09016 08251 06062 08892 08014

FHN-T2 06451 09027 08296 06283 08975 08184

FHN-T3 06461 09176 08541 06386 09137 08448



Polyvore-630 9778 9460 9476

Polyvore-519 9711 9750 9615



































SiameseNet 04993 02808 04997 02731

Bi-LSTM 04992 02817 04990 02739

CSN 05000 02790 04995 02740

FHN 07654 05552 07550 05369

FHN-H 08440 06869 08361 06685











improved











FITB 05998 05780

AUC 08890 08911

NDCG 08211 07970


10568



FHN w Eq (5) 05530 08180 06637 04836 08289 06949

FHN d Eq (4) 05733 08333 07037 05747 08334 07006

FHN w Eq (4) 05062 08463 07274 05578 08243 06717

FHN d Eq (5) 05302 08571 07609 05170 08519 07449

FHN-T3 in Table 2 06461 09176 08541 06386 09137 08448




datasets


































6 Conclusion














representation

10569

References



ICCV 2017









2016

















arXiv 2017



bels In IJCAI 2016






tion In KDD 2017







In CVPR 2016






dia 2014




CVPR 2012





















2014







ECCV 2018




2015



2017





TPAMI 2015








tance In CVPR 2013





AAAI 2016

10570



Train 127326 159729 10712 20230

Test 23054 45505 1944 4437



Train 83416 146475 5133 14594

Test 14654 39085 898 2797



















4 Polyvore Dataset

41 Polyvore-U

















in Table 1

42 Data preparation
















each user















users

















10566

(a) User 1

(b) User 2

(c) User 3














- FHN-T3 set α = 1















10567



(a) SiameseNet [28] 05103 07703 06109 05304 08026 06648

Bi-LSTM [3] 05515 08102 06629 05232 07746 06210

CSN [27] 05536 08187 06744 05617 08215 06703

(b) FHN-T3 (Textual) 05144 08441 07343 04857 08188 06953

FHN-T3 (Visual) 06052 08942 08090 06035 08845 07784

(c) FHN-T0 06066 08989 08213 05770 08761 07821

FHN-T1 06159 09016 08251 06062 08892 08014

FHN-T2 06451 09027 08296 06283 08975 08184

FHN-T3 06461 09176 08541 06386 09137 08448



Polyvore-630 9778 9460 9476

Polyvore-519 9711 9750 9615



































SiameseNet 04993 02808 04997 02731

Bi-LSTM 04992 02817 04990 02739

CSN 05000 02790 04995 02740

FHN 07654 05552 07550 05369

FHN-H 08440 06869 08361 06685











improved











FITB 05998 05780

AUC 08890 08911

NDCG 08211 07970


10568



FHN w Eq (5) 05530 08180 06637 04836 08289 06949

FHN d Eq (4) 05733 08333 07037 05747 08334 07006

FHN w Eq (4) 05062 08463 07274 05578 08243 06717

FHN d Eq (5) 05302 08571 07609 05170 08519 07449

FHN-T3 in Table 2 06461 09176 08541 06386 09137 08448




datasets


































6 Conclusion














representation

10569

References



ICCV 2017









2016

















arXiv 2017



bels In IJCAI 2016






tion In KDD 2017







In CVPR 2016






dia 2014




CVPR 2012





















2014







ECCV 2018




2015



2017





TPAMI 2015








tance In CVPR 2013





AAAI 2016

10570

(a) User 1

(b) User 2

(c) User 3














- FHN-T3 set α = 1















10567



(a) SiameseNet [28] 05103 07703 06109 05304 08026 06648

Bi-LSTM [3] 05515 08102 06629 05232 07746 06210

CSN [27] 05536 08187 06744 05617 08215 06703

(b) FHN-T3 (Textual) 05144 08441 07343 04857 08188 06953

FHN-T3 (Visual) 06052 08942 08090 06035 08845 07784

(c) FHN-T0 06066 08989 08213 05770 08761 07821

FHN-T1 06159 09016 08251 06062 08892 08014

FHN-T2 06451 09027 08296 06283 08975 08184

FHN-T3 06461 09176 08541 06386 09137 08448



Polyvore-630 9778 9460 9476

Polyvore-519 9711 9750 9615



































SiameseNet 04993 02808 04997 02731

Bi-LSTM 04992 02817 04990 02739

CSN 05000 02790 04995 02740

FHN 07654 05552 07550 05369

FHN-H 08440 06869 08361 06685











improved











FITB 05998 05780

AUC 08890 08911

NDCG 08211 07970


10568



FHN w Eq (5) 05530 08180 06637 04836 08289 06949

FHN d Eq (4) 05733 08333 07037 05747 08334 07006

FHN w Eq (4) 05062 08463 07274 05578 08243 06717

FHN d Eq (5) 05302 08571 07609 05170 08519 07449

FHN-T3 in Table 2 06461 09176 08541 06386 09137 08448




datasets


































6 Conclusion














representation

10569

References



ICCV 2017









2016

















arXiv 2017



bels In IJCAI 2016






tion In KDD 2017







In CVPR 2016






dia 2014




CVPR 2012





















2014







ECCV 2018




2015



2017





TPAMI 2015








tance In CVPR 2013





AAAI 2016

10570



(a) SiameseNet [28] 05103 07703 06109 05304 08026 06648

Bi-LSTM [3] 05515 08102 06629 05232 07746 06210

CSN [27] 05536 08187 06744 05617 08215 06703

(b) FHN-T3 (Textual) 05144 08441 07343 04857 08188 06953

FHN-T3 (Visual) 06052 08942 08090 06035 08845 07784

(c) FHN-T0 06066 08989 08213 05770 08761 07821

FHN-T1 06159 09016 08251 06062 08892 08014

FHN-T2 06451 09027 08296 06283 08975 08184

FHN-T3 06461 09176 08541 06386 09137 08448



Polyvore-630 9778 9460 9476

Polyvore-519 9711 9750 9615



































SiameseNet 04993 02808 04997 02731

Bi-LSTM 04992 02817 04990 02739

CSN 05000 02790 04995 02740

FHN 07654 05552 07550 05369

FHN-H 08440 06869 08361 06685











improved











FITB 05998 05780

AUC 08890 08911

NDCG 08211 07970


10568



FHN w Eq (5) 05530 08180 06637 04836 08289 06949

FHN d Eq (4) 05733 08333 07037 05747 08334 07006

FHN w Eq (4) 05062 08463 07274 05578 08243 06717

FHN d Eq (5) 05302 08571 07609 05170 08519 07449

FHN-T3 in Table 2 06461 09176 08541 06386 09137 08448




datasets


































6 Conclusion














representation

10569

References



ICCV 2017









2016

















arXiv 2017



bels In IJCAI 2016






tion In KDD 2017







In CVPR 2016






dia 2014




CVPR 2012





















2014







ECCV 2018




2015



2017





TPAMI 2015








tance In CVPR 2013





AAAI 2016

10570



FHN w Eq (5) 05530 08180 06637 04836 08289 06949

FHN d Eq (4) 05733 08333 07037 05747 08334 07006

FHN w Eq (4) 05062 08463 07274 05578 08243 06717

FHN d Eq (5) 05302 08571 07609 05170 08519 07449

FHN-T3 in Table 2 06461 09176 08541 06386 09137 08448




datasets


































6 Conclusion














representation

10569

References



ICCV 2017









2016

















arXiv 2017



bels In IJCAI 2016






tion In KDD 2017







In CVPR 2016






dia 2014




CVPR 2012





















2014







ECCV 2018




2015



2017





TPAMI 2015








tance In CVPR 2013





AAAI 2016

10570

References



ICCV 2017









2016

















arXiv 2017



bels In IJCAI 2016






tion In KDD 2017







In CVPR 2016






dia 2014




CVPR 2012





















2014







ECCV 2018




2015



2017





TPAMI 2015








tance In CVPR 2013





AAAI 2016

10570

Learning Binary Code for Personalized Fashion Recommendationopenaccess.thecvf.com/content_CVPR_2019/papers/Lu... · 2019-06-10 · Learning Binary Code for Personalized Fashion Recommendation

Documents