INEX 2003, Germany Searching in an XML Corpus Using Content and Structure INEX 2003, Germany Yiftah Ben-Aharon, Sara Cohen, Yael Grumbach, Yaron Kanza,

Searching in an XML CorpusUsing Content and Structure

INEX 2003, GermanyINEX 2003, Germany

Yiftah Ben-Aharon, Sara Cohen, Yael Grumbach, Yaron Kanza, Jonathan Mamou, Yehoshua Sagiv,

Benjamin Sznajder, Efrat Twito

The Hebrew University of Jerusalem

Approach

• IR techniques were extended in the context of XML corpus– The granularity of the retrieval is refined:

fragments of document (and not necessarily whole document) are considered as potential results

– The additional information provided by the structure of the document, and of the query, is exploited when retrieving results

Approach (cont’d)

• An extensible system was built– E.g., new ranking techniques can be added

easily

• The system was implemented in a short time– E.g., topics are translated into XSL stylesheets

• Programming language: Java• Operating System: Windows XP

Topic

• Only the title of the topic, denoted T, is used for retrieval

• We denote– T+ the list of terms in T that are preceded by a

+ sign– T- the list of terms that are preceded by a - sign– To the list of optional terms

• We have implemented our retrieval system only for CO and SCAS topics (not VCAS)

Topic Processor

Filter

Indexer

Extractor

Relevant documents

Ranker Merger

Relevant fragments

Fragments augmented with ranking scores

Topic Result

Indices

IEEE Digital Library

Ranker 5

Ranker 4

Ranker 3

Ranker 2

Ranker 1

Preprocess

• XML documents and topics– Terms are stemmed (using Porter stemmer)– Stopwords are eliminated

• Indices are built

Topic Processor

Filter

Indexer

Extractor

Relevant documents

Ranker Merger

Relevant fragments

Fragments augmented with ranking scores

Topic Result

Indices

IEEE Digital Library

Ranker 5

Ranker 4

Ranker 3

Ranker 2

Ranker 1

Index

• Inverted Keyword Index– Associates each term with the list of documents

(id’s) containing it

• Keyword-Distance Index– Stores information about distance between two

terms over all the sentences in all the documents of the corpus

Index

• Tag Index– Associates to each tag a weight, according to the

“importance” of its content– E.g., the information provided by the front matter is

more important than the information provided by a subsection

• Inverse Document Frequency Index– Associates to each term its IDF, classical in IR– IDF is the fraction of documents in the corpus

containing the term

Topic Processor

Filter

Indexer

Extractor

Relevant documents

Ranker Merger

Relevant fragments

Fragments augmented with ranking scores

Topic Result

Indices

IEEE Digital Library

Ranker 5

Ranker 4

Ranker 3

Ranker 2

Ranker 1

Filter

• Documents not containing all the terms of T+ are considered as irrelevant

• Documents containing all the terms of T+ are extracted from the corpus

Topic Processor

Filter

Indexer

Extractor

Relevant documents

Ranker Merger

Relevant fragments

Fragments augmented with ranking scores

Topic Result

Indices

IEEE Digital Library

Ranker 5

Ranker 4

Ranker 3

Ranker 2

Ranker 1

Relevant Fragments

• Relevant fragments from each document that passed the filtering are extracted

• Relevant fragments– CAS: determined by the topic title– CO: the system determines potentially relevant

fragments• whole document• front matter• abstract• any section• any subsection

Extracting Relevant Fragments from a Document

• An XPath processor is not suitable, since the syntax of CAS topics is more general than that of XPath.

• The relevant fragments are extracted by means of an XSL stylesheet that is generated from T– For CAS topics, the stylesheet also checks that the

returned fragments satisfy the predicates of the title

• The implementation of the translator of topics to XSL stylesheets, is fast

Topic Processor

Filter

Indexer

Extractor

Relevant documents

Ranker Merger

Relevant fragments

Fragments augmented with ranking scores

Topic Result

Indices

IEEE Digital Library

Ranker n

Ranker …

Ranker …

Ranker …

Ranker 1

An Overview of the Ranking Process

• n different rankers give scores based on the structure and the content of the fragments– In our implementation, 5 rankers– For some rankers, the weights of tags are incorporated

into the score– Each ranker gives scores to all the fragments returned

by the extractor– For each result, the scores of all the relevant

fragments are aggregated

Word-Number Ranker

• This ranker counts the number of terms from T- and To appearing in the fragment

• The score is – increased when the number of terms from To is

increased– decreased when the number of terms from T- is

increased

IDF Ranker

• We measure the “rarity” of a term using the classical formula of IDF

• The score is– increased when the number of rare terms from To is

increased– decreased when the number of rare terms from T- is

increased

TFIDF Ranker

• It is an extension of the Vector Space Model to XML documents

• TF counts the number of occurrences of a term in the fragment (and not the whole document)– Each occurrence is multiplied by the weight of its tag

• TFIDF = TF * IDF

• The score of a fragment is computed by– adding the TFIDF of terms from T+ and To– subtracting the TFIDF of terms from T-

Proximity Ranker

• This ranker is based on the correlation between pairs of words from T+ and To appearing in a single phrase in a sliding window containing 5 terms

• Such a pair is called lexical affinity (LA)• The score of a fragment is computed by counting

the number of LA’s• The score is increased when a LA appears under

“important” tags

Similarity Ranker

• Idea: If two terms appear frequently in the same sentence in the corpus, they should be considered as related

• It is a sort of blind query refinement• The score of a fragment is based on

– Distance between the terms of the query and the terms of the fragment

– Increases when the pair appears under “important” tags

Topic Processor

Filter

Indexer

Extractor

Relevant documents

Ranker Merger

Relevant fragments

Fragments augmented with ranking scores

Topic Result

Indices

IEEE Digital Library

Ranker 5

Ranker 4

Ranker 3

Ranker 2

Ranker 1

Merger

• The scores of the various rankers are merged into a single rank

• The main problem is how to determine the relative weight of each ranker

• The scores of the 5 rankers are lexicographically sorted as follows– An order among the rankers is determined– A tuple of the 5 scores is created for each result– The tuples are lexicographically sorted

Merger (cont’d)

• Our submitted results use different orderings of the rankers

• E.g.,– Word Number– Idf– Similarity– Proximity– TFIDF

Conclusion

• Our system builds and uses indices• It combines different rankers• The rankers use both the content and the

structure• The system is extensible

– The implementation uses configuration files– New rankers can be added easily– The system can be easily adapted to changes in

the formal syntax of queries

Future Works

• We still need to experiment thoroughly with the system– Modify the merger by using a single formula to

combine the scores of the different rankers– How to determine the relative weight of each

ranker?– Add and modify rankers

Thank You.

Questions?