A Scalable Semantic Indexing Framework for Peer-to-Peer Information Retrieval University of Illinois at Urbana-Champain Zhichen Xu Yan Chen Northwestern University ACM SIGIR HDIR 2005 Chengxiang Zhai Yahoo! Inc.
A Scalable Semantic Indexing Framework for Peer-to-Peer
Information Retrieval
University of Illinoisat Urbana-Champain
Zhichen XuYan Chen
Northwestern University
ACM SIGIR HDIR 2005
Chengxiang Zhai
Yahoo! Inc.
Motivation
• Rapid information growth requires scalable and robust retrieval architecture
• Problems with centralized retrieval architecture– Hard to maintain freshness of information
– Single-point-of-failure
• Peer-to-Peer IR may be a possible solution – No need for centralized indexing
– Easy to maintain freshness of information
– Resistant to single-point-of-failure
• Challenge: P2P IR architecture?
Term Index vs. Document Index
• Term index– Fast query execution
– Insufficient for supporting sophisticated algorithms such as feedback
– Hard to update (e.g., adding a doc) in a distributed environment
• Document index– Easy to update
– Support advanced retrieval algorithms
– Slow query matching
Previous Work: pSearch [Tang et al. 03]
• Based on document indexing• Address the problem of “slow query
execution” by– Dimension reduction (using LSI)– Exploiting distributed hash tables (DHT)
• Problems– Lack of semantic locality (semantically similar
documents may be stored in quite different nodes)
– Slow index generation– Hard to add a new concept
Proposed Solution: P2PIR, Scalable Semantic Indexing Framework for
IR• Semantic locality
– Achieved through a novel two-phase distributed semantic indexing
– Documents with similar semantics will have indices stored on nearby nodes
• Flexible tradeoff between search accuracy and efficiency
• Support of sophisticated retrieval methods– E.g., feedback and personalized search
• Adaptation to document dynamics– Incrementally incorporate new
documents/concepts
Background on Sample DHTContent-Addressable Network
A B
C D E
• Two key operations• Put (key, object)• Object = get (key)
• Partition Cartesian space into zones
• Each zone is assigned to a computer
• Neighboring zones are routing neighbors
• Object lookup is done through routing
Routing and Location Properties on DHT
• Log(N) hops need to route a key where N is the number of nodes in the overlay
• Log(N) maintenance overhead for routing• Guaranteed success• Fault-tolerant and robust• DoS attack resilient
Becoming increasingly practical for serious use
P2PIR Architecture
• Two stage document indexing– Concept vector generation– Index locator generation and placement
• Open for plugging in “feature extraction”, “relevance ranking” and “query refinement”
XML doc
text doc
featureextraction
structure-awarefeaturevector
termvector
conceptvector construction
indexlocator construction
indexplacementon DHT
queryIndex locator generation
searchon DHT
RelevanceRanking
Applications
P2PIR
P2P DHT system
Internet
Results touser
queryrefinement
Assumptions about the Retrieval Models
• Documents and queries are both represented as vectors – Naturally occurring in the vector-space model– Probabilistic models can be computed as
vector matching as well
• Euclidean distances are reasonably accurate in capturing document topic similarity– Euclidean distances are only used to prune
non-promising documents– Final relevance ranking can be based on more
accurate retrieval functions
Concept Vector Construction
• Group document into k clusters based on the feature vectors
• The centroid of each cluster corresponds to a concept
• Given a document d, the similarity between its feature vector and a concept c (e.g., cosine value between them) defines the weight of d on concept c
• The concept vector of d is composed of its weights on all the concepts
Two-Stage Semantic Indexing
• Stage 1: Fast dimension reduction– Document clustering to identify n*d clusters (d=
DHT dimension)– Represent each document with a vector on this
n*d dimensional space
• Stage 2: Semantic index locator construction– Further partition the n*d clusters into n equal-size
semantically coherent groups, each with size d – Each group forms an index locator (key for
searching DHT)
Fast Dimension Reduction• Regular k-means clustering
– Randomly start with k centroids– Iteratively re-assign documents to each cluster
and re-compute the centroids– Can stop at anytime to obtain rough clusters
• Modification– Start with k relatively different centroids
• Complexity at each iteration: O(kN), where N >>k is the number of documents
• Can be run on a sample of documents• Vector(D)= (sim(D,C1), …, sim(D,Ck))
Index Locator Construction
• Motivation: the dimensionality of concept vectors (e.g., a few hundreds) may be much larger than that of DHT, so hard to place index directly with concept vector
• Basic idea: break the concept vector into multiple chunks with the same dimensionality as that of DHT, and each chunk contains related concepts
• With such division, each document only has a small number of chunks with non-negligible weights for indexing
• Such chunks are called index locators
Index Placement on DHT
• For each index locator of a document d• If its norm (i.e., length of the vector) is
over certain threshold, we put the index locator of d along with its feature vector on the peer node whose DHT address vector matches best with the index locator.
Illustration of Two-Stage Indexing
D1 D2 … DN
C1 C2 … CM
SemanticChunk 1
Semantic Chunk k
Concepts C1 C2 … CM
Doc Di = (x1, x2, …, xd, xd+1, …, x2d, …. …. xM)
Locator 1 Locator 2 ….
(x1, x2, …, xd ) Original vector(D)(x1, x2, …, xd ) Original vector(D’)……
In DHT:
Querying
• Contact any node on the DHT• Project the query vector to find related
concepts, and form the index locators• Use index locators to route to DHT nodes
with the indices and feature vectors of related documents
• Use original query vector and document vectors to perform relevance ranking
• This local retrieval process can expand to neighboring DHT nodes until enough relevant results have been identified
Adaptation to Corpus Dynamics
• Basic idea: Incrementally add new documents/concepts without affecting existing indices, and periodically (very infrequently) rebuild index locators for all documents
• When a set of new documents emerge, we check1. whether they contain new frequently-used terms
or new heavy weighted terms2. whether their concept vectors belong to any
existing cluster in the existing semantic space
Adaptation to Corpus Dynamics (II)
• To add and index a new concept c– If c belongs to an existing concept chunk whose
size is less than that of the underlying DHT, we can add c to that cluster by using the next available entry of the index locator.
– Otherwise, we generate a new concept group and a new set of index locators to represent c
• Generate the index locators for the new documents, and deploy their indices on DHT
• Finally, multicast the addition of the new concept c, and the addition of new concept group to all DHT nodes, so that they can route queries about c
Example for Corpus Dynamics
• When new documents on “Bin Larden” appear, we detect it as a new concept relating to the concept group “terrorism”.
• If the dimensionality of DHT is 20, and the size of “terrorism” concept group is 17– Just add “Bin Larden” to that group as dimension
18 of the index locator. – The corresponding index locators of existing
documents have weight zero as default on dimension 18, and thus remain the same.
• Otherwise, the terrorism concept group already full, we generate a new concept group for “Bin Larden” (i.e., a new set of index locators).
Summary
• Propose a scalable semantic indexing framework for peer-to-peer information retrieval: P2PIR– Index placement with good semantic locality,
leading to good retrieval accuracy and efficiency
– Tunable framework and flexibility– Incremental adaptation to document/concept
dynamics
• Prototype and evaluation of P2PIR in progress