Ranking Preserving Hashing for Fast Similarity Search · tags). Locality-Sensitive Hashing (LSH) [Datar et al., 2004] ... ing results, very limited work explores the ranking accuracy,

Ranking Preserving Hashing for Fast Similarity Search

Qifan Wang, Zhiwei Zhang and Luo SiComputer Science Department, Purdue University

West Lafayette, IN 47907, [email protected], [email protected], [email protected]

AbstractHashing method becomes popular for large scalesimilarity search due to its storage and compu-tational efficiency. Many machine learning tech-niques, ranging from unsupervised to supervised,have been proposed to design compact hashingcodes. Most of the existing hashing methodsgenerate binary codes to efficiently find similardata examples to a query. However, the rankingaccuracy among the retrieved data examples isnot modeled. But in many real world applications,ranking measure is important for evaluating thequality of hashing codes. In this paper, we pro-pose a novel Ranking Preserving Hashing (RPH)approach that directly optimizes a popular rankingmeasure, Normalized Discounted Cumulative Gain(NDCG), to obtain effective hashing codes withhigh ranking accuracy. The main difficulty inthe direct optimization of NDCG measure isthat it depends on the ranking order of dataexamples, which forms a non-convex non-smoothoptimization problem. We address this challengeby optimizing the expectation of NDCG measurecalculated based on a linear hashing function. Agradient descent method is designed to achieve thegoal. An extensive set of experiments on two largescale datasets demonstrate the superior rankingperformance of the proposed approach over severalstate-of-the-art hashing methods.

1 IntroductionSimilarity search is an important problem in many machinelearning applications. The purpose of similarity search is toidentify similar data examples to a given query example.Due to the explosive growth of data on the Internet, a hugeamount of data has been generated, which indicates that itis important to design efficient solutions of similarity searchfor large scale data. Traditional similarity search methodsare difficult to be used directly for large scale data sincecomputing the similarity using the original features (usuallyin high dimensional space) exhaustively between the queryexample and every candidate example is impractical for largeapplications. Recently, hashing methods [Liu et al., 2013;

Kong and Li, 2012a; Wang et al., 2014a; Zhang and Li, 2014;Bergamo et al., 2011; Kong and Li, 2012b; Xia et al., 2014;Rastegari et al., 2013; Lin et al., 2014; Wang et al., 2014c;Zhai et al., 2013; Wang et al., 2013c] have been proposed forfast similarity search in many large scale problems includingdocument retrieval [Wang et al., 2013b], object recognition[Torralba et al., 2008], image matching [Strecha et al., 2012],etc. These hashing methods design compact binary codein a low-dimensional space for each data example so thatsimilar data examples are mapped to similar binary codes.In the retrieval process, these hashing methods first transformeach query example into its corresponding binary code. Thensimilarity search can be simply conducted by calculatingthe Hamming distances between the codes of available dataexamples and the query, and selecting data examples withinsmall Hamming distances. In this way, data examples areencoded and highly compressed within a low-dimensionalbinary space, which can usually be loaded in main memoryand stored efficiently. The retrieval process can be conductedefficiently as the Hamming distance between two codes issimply the number of bits that differ and can be calculatedusing bitwise operation XOR. Existing hashing methodscan be divided into two groups: unsupervised and semi-supervised/supervised.

Unsupervised hashing methods generate hashing codeswithout the requirement of supervised information (e.g.,tags). Locality-Sensitive Hashing (LSH) [Datar et al., 2004]is one of the most popular methods, which simply usesrandom linear projections to map data examples from a highdimensional Euclidean space to a low-dimensional binaryspace. The work in [Kulis and Grauman, 2009] extended LSHby exploiting kernel similarity for better retrieval efficacy.The Principle Component Analysis (PCA) Hashing [Lin etal., 2010] method utilize the coefficients from the top kprincipal components to represent each example, and thecoefficients are further binarized using the median value.Recently, Spectral Hashing (SH) [Weiss et al., 2008] isproposed to design compact binary codes with balanced anduncorrelated constraints. Isotropic Hashing (IsoHash) [Kongand Li, 2012b] tries to learn an orthogonal matrix to makethe data variance as equal as possible along each projectiondimension. The work in [Wang et al., 2015] proposes to learnthe binary codes on structured data.

Semi-supervised or supervised hashing methods utilize

Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 2015)

3911

some supervised information such as semantic labels forgenerating effective hashing codes. Iterative Quantization(ITQ) method has been proposed in [Gong et al., 2012] thattreats the content features and tags as two different views, andthe hashing codes are then learned by extracting a commonspace from these two views. This method has been extendedto multi-view hashing [Gong et al., 2013]. A semi-supervisedhashing (SSH) method is proposed in [Wang et al., 2010]which utilizes pairwise knowledge between data examplesbesides their content features for learning more effectivehashing codes. A kernelized supervised hashing (KSH)framework proposed in [Liu et al., 2012] imposes pairwiserelationship between data examples to obtain hashing codes.More recently, a ranking-based supervised hashing (RSH)[Wang et al., 2013a] method is proposed to leverage listwiseranking information to preserve the ranking order.

Although existing hashing methods have achieved promis-ing results, very limited work explores the ranking accuracy,which is important for evaluating the quality of hashing codesin real world applications. Consider the following scenario:given a query example xq and three relevant/similar dataexamples x1, x2, x3 but with different relevance values asr1 > r2 > r3 to the query. Most existing hashing methodsonly model the relevance of a data example to a query ina binary way, i.e., each example is either relevant to thequery or irrelevant. In other words, these methods treat x1,x2 and x3 as relevant examples to xq with no difference.But in practice it will be more desirable if x1 could bepresented before x2 and x3 since it is more relevant to xqthan the other two. Some ranking based hashing methods[Wang et al., 2013a; Yagnik et al., 2011; Zhang et al., 2013]have been recently proposed to improve the hashing codeperformance by modeling the ranking order with respect torelevance values. However, these methods do not differentiatethe situations where (r1, r2, r3) = (3, 2, 1) and (r1, r2, r3) =(10, 2, 1) due to their identical ranking orders, i.e., r1 >r2 > r3. But ideally, the Hamming distance between thelearned hashing codes of x1 and xq should be smaller inthe later situation than in the former one since the relevancevalue of x1 to xq is much larger in the later situation (10versus 3). Therefore, these methods may fail to preserve thespecific relevance values in the learned hashing codes, whilethe relevance values are important in evaluating the searchaccuracy.

This paper proposes a novel Ranking Preserving Hashing(RPH) approach that directly optimizes the popular rankingaccuracy measure, Normalized Discounted Cumulative Gain(NDCG), to learn effective ranking preserving hashing codesthat not only preserves the ranking order but also modelsthe relevance values of data examples to the queries in thetraining data. The main difficulty in direct optimization ofNDCG is that it depends on the rankings of data examplesrather than their hashing codes, which forms a non-convexnon-smooth objective. We then address this challenge byoptimizing the expectation of NDCG measure calculatedbased on a linear hashing function to convert the problem intoa smooth and differentiable optimization problem. A gradientdescent method is applied to solve this relaxed problem. Weconduct an extensive set of experiments on two large scale

datasets of both images and texts to demonstrate the superiorsearch accuracy of the proposed approach over several state-of-the-art hashing methods.

2 Ranking Preserving Hashing2.1 Approach OverviewThe proposed Ranking Preserving Hashing (RPH) approachvia optimizing NDCG measure mainly contains three ingre-dients as shown in Figure 1: (1) Ground-truth relevance listto a query, which is constructed from the training data (theleft part in Fig.1). (2) Ranking positions of data examplesto a query, which are computed based on the hashing codes(the right part in Fig.1). (3) NDCG value, which measuresthe consistency between the ground-truth relevance list andthe calculated ranking positions (the middle part in Fig.1).In other words, the more the hashing codes agree with therelevance list, the higher the NDCG value will be. Then theranking preserving hashing codes are learned by optimizingthe NDCG measure on the training data.

2.2 Problem StatementWe first introduce the problem of RPH. Assume thereare n data examples in the dataset, denoted as: XXX ={x1, x2, . . . , xn} ∈ Rd×n, where d is the dimensionalityof the features. In addition, there is a query set QQQ ={q1, q2, . . . , qm} and for each query example qj , we have arelevance list of nj data examples from XXX , which can bewritten as:

r(qj ,XXX) = (rj1, rj2, . . . , r

jnj ) (1)

where each element rji represents the relevance of dataexample xji to the query qj . If rju > rjv , it indicates that dataexample xju is more relevant or more similar to qj than xjv andxju should rank higher than xjv . The goal is to obtain a linearhashing function f : Rd → {−1, 1}B , which maps each dataexample xi to its binary hashing code ci (B is the number ofhashing bits) to maximize the search/ranking accuracy. Thelinear hashing function is defined as:

ci = f(xi) = sgn(WWWxi) (2)

where WWW ∈ RB×d is the coefficient matrix representing thehashing function and sgn is the sign function. ci ∈ {−1, 1}Bis the binary hashing code of xi.

Note that the ground-truth relevance list can be easilyobtained if a relevance measure between data examples ispredefined, e.g., l2 distance in Euclidean space. On the otherhand, if given the semantic label/tag information, it is alsofairly straightforward to convert semantic labels to relevancevalues through counting the number of shared labels betweenthe query and the data example.

2.3 Problem FormulationHashing methods are popularly used for large scale similaritysearch. As aforementioned, most of existing hashing methodsonly focus on retrieving all relevant or similar data examplesto a given query without exploring the ranking accuracy.

3912

Figure 1: An overview of the proposed RPH approach.

However, in many real world applications, it is desirable andimportant to present a more relevant example to a query infront of a less relevant one. Different from existing hashingmethod, in this work, we propose to learn ranking preservinghashing codes that not only retrieve all possible relevantexamples but at the same time preserve their rankings basedon their relevance values to the query.

Given the binary hashing codes, the ranking positions ofdata examples to a query q are determined by the Hammingdistances between their hashing codes and the query code.Specifically, if a data example is similar or relevant to aquery, then their Hamming distance should be small. In otherwords, the higher the rank of a data example to a query, thesmaller the Hamming distance between the hashing codes is.The Hamming distance between two binary hashing codes isgiven by the number of bits that are different between themand can be calculated as:

Ham(cq, ci) =1

4‖cq − ci‖2 =

1

2(B − cTq ci) (3)

Then the ranking position π(xi) can be calculated as:

π(xi) = 1 +

n∑k=1

I (Ham(cq, ci) > Ham(cq, ck))

= 1 +n∑

k=1

I(cTq (ck − ci) > 0

) (4)

where I(s) is the indicator function that outputs 1 whenstatement s is true and 0 otherwise. Intuitively, the rankingposition of a data example to a query is equivalent to 1 plusthe number of data examples whose hashing codes are closerto the query code.

In order to achieve high ranking quality hashing codes, wewant the ranking positions calculated in the Hamming spacein Eqn.4 to be consistent with the ground-truth relevance listin Eqn.1. Then a natural question to ask is how to measurethe ranking consistency? In this paper, we use a well-knownmeasure, Normalized Discounted Cumulative Gain (NDCG)[Qin et al., 2010; Wang et al., 2013d] which is widelyapplied in many information retrieval and machine learning

applications, to evaluate the ranking consistency as:

NDCG =1

Z

n∑i=1

2rπ−1(i) − 1

log(1 + i)=

1

Z

n∑i=1

2ri − 1

log(1 + π(xi))

(5)

where Z is the normalization factor so that the maximumvalue of NDCG is 1, which can be calculated by rankingthe examples based on their relevance to the query. π(xi)is the ranking position of xi to the query based on theHamming distance of their hashing codes and π−1(i)denotes the data example at i-th ranking position. ri is thecorresponding relevance value. 1

log(1+i) can be viewed as theweight of the i-th rank data example, which indicates thatNDCG emphasizes the importance of the higher ranked dataexamples than those examples with lower ranks. Therefore,NDCG is usually truncated at a particular rank level (e.g.,top K retrieved examples) instead of all n examples. Fromthe above definition of NDCG, it can be seen that the largerthe NDCG value is, the more the hashing codes agree withthe relevance list, and the maximal NDCG value is obtainedwhen the ranking positions of data examples are completelyconsistent with their relevance values to the query. Byoptimizing the NDCG measure, the learned hashing functionnot only preserves the ranking order of the data examples butalso ensures that the hashing codes are consistent with therelevance values in the training data. Then the entire objectiveis to minimize the negative summation of NDCG values on alltraining queries:

J(WWW ) = −m∑j=1

1

Zj

nj∑i=1

2rji − 1

log(1 + πj(xji ))

(6)

Directly minimizing the objective function in Eqn.6 isintractable since it depends on the ranking positions of dataexamples (Eqn.4), resulting in a non-convex non-smoothoptimization problem. We then address this challenge byusing the expectation of ranking position π̂j(x

ji ) instead of

3913

πj(xji ) as:

π̂j(xji ) = 1 + E

[n∑

k=1

I(cTqj (ck − ci) > 0)

]

= 1 +n∑

k=1

Pr(cTqj (ck − ci) > 0

) (7)

where Pr(cTqj (ck − ci) > 0) means the probability thatthe ranking position of data example xk is higher than theposition of xi to query qj and we use a logistic function tomodel this probability as:

Pr(cTqj (ck − ci) > 0

)=

1

1 + exp(−cTqj (ck − ci))

=1

1 + exp(−sgn(WWWqj)T (sgn(WWWxk)− sgn(WWWxi)))

(8)

The motivation of the derivation in Eqn.7 and Eqn.8 isthat we approximate the intractable optimization for NDCGwith a tractable probabilistic framework. Firstly, the rankingposition of each data example can be calculated exactlybased on Eqn.4. However, due to the intractability, we modelthe problem in a probabilistic framework by computing theexpectation of the ranking position. The using of expectationto represent the true ranking position is widely adoptedin learning to rank approaches due to its good probabilityapproximation and computational tractability. Secondly, theusing of logistic function in Eqn.8 to model the probability isbased on the intuition that a data example should be rankedhigher if its hashing code is closer to the query. There arealso other alternatives to model the probability. Due to thepopularity of logistic function used in learning to rank, weadopt it in our formulation.

The above probability function is still non-differentiablewith respect to WWW due to the embedded sign function.Therefore, as suggested in [Wang et al., 2013a; 2014b], wedrop off the sign function and use the signed magnitude inthe probability function as:

Pr(cTqj (ck − ci) > 0

)=

1

1 + exp(−qTj WWWTWWW (xk − xi))(9)

By substituting the expected ranking position into theNDCG measure, the final objective in Eqn.6 can be rewrittenas:

min J(WWW ) = −m∑j=1

1

Zj

nj∑i=1

2rji − 1

log(1 + π̂j(xji ))

s.t. WWWWWWT = III

(10)

where WWWWWWT = III is the orthogonality constraint whichensures the learned hashing codes to be uncorrelated witheach other and hold least redundant information.

2.4 OptimizationWe first convert the hard constraint into a soft penalty termby adding a regularizer to the objective. The reason is that

most of the variance is contained in a few top projectionsfor many real world datasets. The orthogonality constraintforces hashing methods to choose those directions with verylow variance progressively, which may substantially reducethe quality of hashing codes. This issue is also pointed outin [Liu et al., 2012; Wang et al., 2012]. Therefore, insteadof adding hard orthogonality constraint, we impose a softorthogonality/penalty term as:

J(WWW ) = −m∑j=1

1

Zj

nj∑i=1

2rji − 1

log(1 + π̂j(xji ))

+ α‖WWWWWWT − III‖2F

(11)

where α is a trade-off parameter to balance the weightsbetween the two terms. Although the objective in Eqn.11is still non-convex, it is smooth and differentiable whichenables gradient descent methods to be applied for efficientoptimization. The gradients of the two terms with respect toWWW are given below:

dπ̂j(xji )

dWWW=

nj∑k=1

exp(−qTj WWWTWWW (xk − xi))

WWW((xk − xi)qTj + qj(xk − xi)T

)(1 + exp(−qTj WWWTWWW (xk − xi)))2

(12)

d‖WWWTWWW − III‖2FdWWW

= 4WWWT (WWWWWWT − III) (13)

Then the gradient of dJ(WWW )dWWW can be computed by combining

the above two gradients with some additional mathematicalcalculation. With this obtained gradient, L-BFGS quasi-Newton method [Liu and Nocedal, 1989] is applied tosolve the optimization problem. The full RPH approach issummarized in Algorithm 1.

Algorithm 1 Ranking Preserving Hashing (RPH)Input: Training examplesXXX , query examplesQQQ and param-

eters α.Output: Hashing functionWWW and hashing codesCCC.

1: Compute the relevance vector rji in Eqn.1.2: InitializeWWW .3: repeat Gradient Descent4: Compute the gradient in Eqn.12.5: Compute the gradient in Eqn.13.6: UpdateWWW by optimizing the objective function.7: until the solution converges8: Compute the hashing codesCCC using Eqn.2.

2.5 DiscussionThe idea of modeling the NDCG measure to maximizethe search/ranking accuracy is also utilized in learningto rank [Valizadegan et al., 2009; Weimer et al., 2007].However, these learning to rank methods are not based onbinary hashing codes, but on learning effective documentpermutation. Unlike in our formulation, the NDCG measuremodeled in learning to rank methods does not involve linear-projection based hashing function, on which the ranking

3914

NUSWIDE Flickr1mMethods NDCG@5 NDCG@10 NDCG@20 NDCG@5 NDCG@10 NDCG@20

RPH 0.2570.2570.257 0.2490.2490.249 0.2340.2340.234 0.3130.3130.313 0.2980.2980.298 0.2830.2830.283RSH 0.242 0.238 0.226 0.288 0.271 0.259KSH 0.223 0.217 0.198 0.265 0.252 0.237SSH 0.216 0.209 0.195 0.251 0.242 0.230SH 0.193 0.185 0.172 0.250 0.234 0.221

Table 1: Results of NDCG@K using Hamming Ranking on both datasets, with 64 hashing bits.

position is determined. Moreover, we need to find theexpected ranking position of each data example according tothe Hamming distance between the hashing codes, which isvery different to learning to rank methods.

The learning algorithm of RPH for deriving the optimalhashing function is fairly fast. During each iteration of thegradient descent method, we need to compute the gradients inEqns.12 and 13, which involves some matrix multiplications.The complexity for calculating the gradient in Eqn.12 isbounded by O(mnjdB) since bothWWW (xk−xi) andWWW (xk−xi)q

Tj requires O(dB). The complexity for calculating the

gradient in Eqn.13 is simply O(d2B) which only involvesWWWWWWT . Therefore, the total complexity of each iteration ofthe gradient descent method is O(mn̂dB + d2B) and thelearning algorithm is fairly scalable since its time complexityis linear in the number of training queries m and the averagenumber of data examples n̂ associated with each query.

3 Experiment3.1 Datasets and SettingWe evaluate proposed research on two image benchmarks:NUSWIDE and Flickr1m, which have been widely used inthe evaluation of hashing methods. NUSWIDE1 [Chua et al.,2009] is created by NUS lab for evaluating image retrievaltechniques. It contains 270k images associated with about5k different tags. We use a subset of 110k image exampleswith the most common 1k tags in our experiment. Flickr1m2

[Huiskes et al., 2010] is collected from Flicker images forimage annotation and retrieval tasks. This dataset contains1 million image examples associated with more than 7kunique semantic tags. A subset of 250k image examples withthe most common 1k tags is used in our experiment. 512-dimensional GIST descriptors [Oliva and Torralba, 2001] areused as image features. Since both datasets are associatedwith multiple semantic labels/tags, the ground-truth relevancevalues can be naturally derived based on the number of sharedsemantic labels between data examples.

We implement our algorithm using Matlab on a PC withIntel Duo Core i5-2400 CPU 3.1GHz and 8GB RAM. Theparameter α is tuned by cross validation through the grid{0.01, 0.1, 1, 10, 100} and we will discuss more details onhow it affects the performance of our approach later. Foreach experiment, we randomly choose 1k examples as testingqueries. Within the remaining data examples, we randomly

1http://lms.comp.nus.edu.sg/research/NUS-WIDE.htm2http://press.liacs.nl/mirflickr/

sample 500 training queries and for each query, we randomlysample 1000 data examples to construct the ground-truthrelevance list. We will discuss the performance with differentnumber of training queries later in our experiments. Finally,we repeat each experiment 10 times and report the resultbased on the average over the 10 runs.

The proposed RPH approach is compared with fourdifferent hashing methods, including Spectral Hashing (SH)[Weiss et al., 2008], Semi-Supervised Hashing (SSH) [Wanget al., 2010], Kernel Supervised Hashing (KSH) [Liu et al.,2012] and Ranking-based Supervised Hashing (RSH) [Wanget al., 2013a]. SH is an unsupervised method and does notuse any label information. We use the standard settings in[Weiss et al., 2008] in our experiments. For SSH and KSH,we randomly sample 2k data examples and use their ground-truth labels to generate pairwise similarity matrix as part ofthe training data. Gaussian RBF kernel is used in KSH. To geta fair comparison, for RSH, we randomly sample 500 queryexamples and 1000 data examples to compute the ground-truth ranking lists.

3.2 Evaluation MetricsTo conduct fair evaluation, we follow two criteria which arecommonly used in the literature: Hamming Ranking andHash Lookup. Hamming Ranking ranks all the candidateexamples according to their Hamming distance from thequery and the top K examples are returned as the desiredneighbors. We use NDCG@K to evaluate the ranking qualityof the top K retrieved examples. Hash Lookup returnsall the examples within a certain Hamming radius r of thequery. We use average cumulative gain (ACG) to measurethe quality of these returned examples, which is calculatedas: ACGr = 1

|Nr|∑

xi∈Nr ri, where Nr is the set of theretrieved data examples within a Hamming radius r and riis the relevance value of a retrieved data example xi. Ahamming radius of r = 2 is used to retrieve the neighborsin the experiments.

3.3 Results and DiscussionWe first report the results of NDCG@5, NDCG@10 andNDCG@20 of different hashing methods using HammingRanking on two datasets with 64 hashing bits in Table1. From these comparison results, it can be seen that RPHgives the overall best performance among all five hashingmethods on both datasets. For example, the performance ofour method boosts about 4.6% on NUSWIDE dataset, with9.9% improvement on Flickr1m dataset compared to RSH

3915

Figure 2: Performance evaluation on both datasets withdifferent number of hashing bits. (a)-(b): NDCG@10 usingHamming Ranking. (c)-(d): ACG with Hamming radius 2using Hash Lookup.

under NDCG@10 measure. We can see from Table 1 thatSH does not perform well in all cases. This is because SHis an unsupervised hashing method which does not utilizeany supervised information into learning hashing codes. Formethods SSH and KSH, they both achieve better resultsthan SH since these methods incorporate some pairwiseknowledge between data examples in addition to the contentfeatures for learning effective hashing codes. However, theranking order is not preserved in the learned hashing codesof these two methods and thus, the ranking-based supervisedhashing method RSH which models the listwise rankinginformation can generate more accurate hashing codes withlarger NDCG values than SSH and KSH. On the other hand,RPH method substantially outperforms RSH since it directlyoptimizes the NDCG measure to learn high quality hashingcodes that not only preserve the ranking order but alsopreserve the relevance values of data examples to the queryin the training data. Therefore, the search/ranking accuracycan be maximized which is coincides with our expectation.

The second set of experiments evaluate the performanceof different hashing methods by varying the number ofhashing bits in the range of {16, 32, 64, 128, 256}. Theresults of NDCG@10 using Hamming Ranking on bothdatasets are reported in Fig.2(a)-(b), with the ACG results ofHamming radius 2 using Hash Lookup shown in Fig.2(c)-(d). Not surprisingly, from Fig.2(a)-(b) we can see that theperformance of different methods improves when the numberof hashing bits increases from 16 to 256 and our RPHmethod outperforms the other compared hashing methodswhich is consistent with the results in Table 1. However,we can also observe from Fig.2(c)-(d) that the ACG result

Figure 3: Parameter sensitivity results of NDCG@10 on bothdatasets with 64 hashing bits.

of most compared methods decreases when the number ofhashing bits increases after 64. The reason is that whenusing longer hashing bits, the Hamming space becomesincreasingly sparse and very few data examples fall withinthe Hamming ball of radius 2, resulting in many querieswith empty returns (we count the ACG as zero in this case).Similar behavior is also observed in [Wang et al., 2013a;2014b]. In this situation, however, the ACG results of RPHare still consistently better than other baselines.

To prove the robustness of the proposed method, weconduct parameter sensitivity experiments on both datasets.In each experiment, we tune the trade-off parameter α fromthe grid {0.5, 1, 2, 4, 8, 32, 128}. We report the results ofNDCG@10 with 64 hashing bits in Fig.3. It is clear fromthese experimental results that the performance of RPHis relatively stable with respect to α in a wide range ofvalues. The results also prove that using soft penalty with anappropriate weight parameter is better than enforcing the hardorthogonality constraint (corresponds to infinite α).

4 ConclusionThis paper proposes a novel Ranking Preserving Hashing(RPH) approach that directly optimizes the ranking accuracymeasure, Normalized Discounted Cumulative Gain (NDCG).We handle the difficulty of non-convex non-smooth optimiza-tion by using the expectation of NDCG measure calculatedbased on the linear hashing function and then solve therelaxed smooth optimization problem with a gradient descentmethod. Experiments on two large scale datasets demonstratethe superior performance of the proposed approach overseveral state-of-the-art hashing methods. In future, we planto investigate generalization error bound for the proposedlearning method. We also plan to apply some sequentiallearning approach to accelerate the training speed.

5 AcknowledgmentsThis work is partially supported by NSF research grants IIS-0746830, DRL-0822296, CNS-1012208, IIS-1017837, CNS-1314688 and a research grant from Office of Naval Research(ONR-11627465). This work is also partially supported bythe Center for Science of Information (CSoI), an NSFScience and Technology Center, under grant agreement CCF-0939370.

3916

References[Bergamo et al., 2011] Alessandro Bergamo, Lorenzo Torresani,

and Andrew W. Fitzgibbon. Picodes: Learning a compact codefor novel-category recognition. In NIPS, pages 2088–2096, 2011.

[Chua et al., 2009] Tat-Seng Chua, Jinhui Tang, Richang Hong,Haojie Li, Zhiping Luo, and Yantao Zheng. Nus-wide: a real-world web image database from national university of singapore.In CIVR, 2009.

[Datar et al., 2004] Mayur Datar, Nicole Immorlica, Piotr Indyk,and Vahab S. Mirrokni. Locality-sensitive hashing schemebased on p-stable distributions. In Symposium on ComputationalGeometry, pages 253–262, 2004.

[Gong et al., 2012] Yunchao Gong, Svetlana Lazebnik, AlbertGordo, and Florent Perronnin. Iterative quantization: Aprocrustean approach to learning binary codes for large-scaleimage retrieval. IEEE TPAMI, 2012.

[Gong et al., 2013] Yunchao Gong, Qifa Ke, Michael Isard, andSvetlana Lazebnik. A multi-view embedding space for modelinginternet images, tags, and their semantics. IJCV, 2013.

[Huiskes et al., 2010] Mark J. Huiskes, Bart Thomee, andMichael S. Lew. New trends and ideas in visual conceptdetection: the mir flickr retrieval evaluation initiative. InMultimedia Information Retrieval, pages 527–536, 2010.

[Kong and Li, 2012a] Weihao Kong and Wu-Jun Li. Double-bitquantization for hashing. In AAAI, 2012.

[Kong and Li, 2012b] Weihao Kong and Wu-Jun Li. Isotropichashing. In NIPS, pages 1655–1663, 2012.

[Kulis and Grauman, 2009] Brian Kulis and Kristen Grauman.Kernelized locality-sensitive hashing for scalable image search.In ICCV, pages 2130–2137, 2009.

[Lin et al., 2010] Ruei-Sung Lin, David A. Ross, and Jay Yagnik.Spec hashing: Similarity preserving algorithm for entropy-basedcoding. In CVPR, pages 848–854, 2010.

[Lin et al., 2014] Guosheng Lin, Chunhua Shen, and Jianxin Wu.Optimizing ranking measures for compact binary code learning.In ECCV, pages 613–627, 2014.

[Liu and Nocedal, 1989] Dong C. Liu and Jorge Nocedal. Onthe limited memory bfgs method for large scale optimization.Mathematical Programming, 45:503–528, 1989.

[Liu et al., 2012] Wei Liu, Jun Wang, Rongrong Ji, Yu-Gang Jiang,and Shih-Fu Chang. Supervised hashing with kernels. In CVPR,pages 2074–2081, 2012.

[Liu et al., 2013] Xianglong Liu, Junfeng He, and Bo Lang.Reciprocal hash tables for nearest neighbor search. In AAAI,2013.

[Oliva and Torralba, 2001] Aude Oliva and Antonio Torralba. Mod-eling the shape of the scene: A holistic representation of thespatial envelope. IJCV, 42(3):145–175, 2001.

[Qin et al., 2010] Tao Qin, Tie-Yan Liu, and Hang Li. A generalapproximation framework for direct optimization of informationretrieval measures. Inf. Retr., 13(4):375–397, 2010.

[Rastegari et al., 2013] Mohammad Rastegari, Jonghyun Choi,Shobeir Fakhraei, Daume Hal, and Larry S. Davis. Predictabledual-view hashing. In ICML (3), pages 1328–1336, 2013.

[Strecha et al., 2012] Christoph Strecha, Alexander A. Bronstein,Michael M. Bronstein, and Pascal Fua. Ldahash: Improvedmatching with smaller descriptors. IEEE TPAMI, 34(1):66–78,2012.

[Torralba et al., 2008] Antonio Torralba, Robert Fergus, andWilliam T. Freeman. 80 million tiny images: A large data setfor nonparametric object and scene recognition. IEEE TPAMI,30(11):1958–1970, 2008.

[Valizadegan et al., 2009] Hamed Valizadegan, Rong Jin, RuofeiZhang, and Jianchang Mao. Learning to rank by optimizing ndcgmeasure. In NIPS, pages 1883–1891, 2009.

[Wang et al., 2010] Jun Wang, Ondrej Kumar, and Shih-Fu Chang.Semi-supervised hashing for scalable image retrieval. In CVPR,pages 3424–3431, 2010.

[Wang et al., 2012] Jun Wang, Sanjiv Kumar, and Shih-Fu Chang.Semi-supervised hashing for large-scale search. IEEE TPAMI,34(12):2393–2406, 2012.

[Wang et al., 2013a] Jun Wang, Wei Liu, Andy Sun, and Yu-GangJiang. Learning hash codes with listwise supervision. In ICCV,2013.

[Wang et al., 2013b] Qifan Wang, Dan Zhang, and Luo Si. Seman-tic hashing using tags and topic modeling. In SIGIR, pages 213–222, 2013.

[Wang et al., 2013c] Qifan Wang, Dan Zhang, and Luo Si.Weighted hashing for fast large scale similarity search. In CIKM,pages 1185–1188, 2013.

[Wang et al., 2013d] Yining Wang, Liwei Wang, Yuanzhi Li,Di He, and Tie-Yan Liu. A theoretical analysis of ndcg typeranking measures. In COLT, pages 25–54, 2013.

[Wang et al., 2014a] Qifan Wang, Bin Shen, Shumiao Wang, LiangLi, and Luo Si. Binary codes emmbedding for fast image taggingwith incomplete labels. In ECCV, 2014.

[Wang et al., 2014b] Qifan Wang, Luo Si, and Dan Zhang. Learn-ing to hash with partial tags: Exploring correlation between tagsand hashing bits for large scale image retrieval. In ECCV, pages378–392, 2014.

[Wang et al., 2014c] Qifan Wang, Luo Si, Zhiwei Zhang, and NingZhang. Active hashing with joint data example and tag selection.In SIGIR, 2014.

[Wang et al., 2015] Qifan Wang, Luo Si, and Bin Shen. Learningto hash on structured data. In AAAI, 2015.

[Weimer et al., 2007] Markus Weimer, Alexandros Karatzoglou,Quoc V. Le, and Alex J. Smola. Cofi rank - maximum marginmatrix factorization for collaborative ranking. In NIPS, 2007.

[Weiss et al., 2008] Yair Weiss, Antonio Torralba, and RobertFergus. Spectral hashing. In NIPS, pages 1753–1760, 2008.

[Xia et al., 2014] Rongkai Xia, Yan Pan, Hanjiang Lai, Cong Liu,and Shuicheng Yan. Supervised hashing for image retrieval viaimage representation learning. In AAAI, pages 2156–2162, 2014.

[Yagnik et al., 2011] Jay Yagnik, Dennis Strelow, David A. Ross,and Ruei-Sung Lin. The power of comparative reasoning. InICCV, pages 2431–2438, 2011.

[Zhai et al., 2013] Deming Zhai, Hong Chang, Yi Zhen, XianmingLiu, Xilin Chen, and Wen Gao. Parametric local multimodalhashing for cross-view similarity search. In IJCAI, 2013.

[Zhang and Li, 2014] Dongqing Zhang and Wu-Jun Li. Large-scale supervised multimodal hashing with semantic correlationmaximization. In AAAI, pages 2177–2183, 2014.

[Zhang et al., 2013] Lei Zhang, Yongdong Zhang, Jinhui Tang,Ke Lu, and Qi Tian. Binary code ranking with weighted hammingdistance. In CVPR, pages 1586–1593, 2013.

3917