Information Retrieval Document Parsing
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Information Retrieval
Document Parsing
Basic indexing pipeline
Tokenizer
Token stream. Friends Romans Countrymen
Linguistic modules
Modified tokens. friend roman countryman
Indexer
Inverted index.
Documents tobe indexed.
Friends, Romans, countrymen.
Parsing a document
What format is it in? pdf/word/excel/html?
What language is it in? What character set is in use?
Plain ASCII, UTF-8, UTF-16,…
Each of these is a classification problem, with many complications…
Tokenization: Issues
Chinese/Japanese no spaces between words: Not always guaranteed a unique tokenization Dates/amounts in multiple formats
フォーチュン 500 社は情報不足のため時間あた $500K( 約 6,000 万円 )
Katakana Hiragana Kanji “Romaji”
What about DNA sequences ? ACCCGGTACGCAC...
Definition of Tokens What you can search !!
Case folding
Reduce all letters to lower case exception: upper case (in mid-
sentence?) e.g., General Motors USA vs. usa
Morgen will ich in MIT … Is this the
German “mit”?
Stemming
Reduce terms to their “roots” language dependent
e.g., automate(s), automatic, automation all reduced to automat.
e.g., casa, casalinga, casata, casamatta, casolare, casamento, casale, rincasare, case reduced to cas
Porter’s algorithm
Commonest algorithm for stemming English Conventions + 5 phases of reductions
phases applied sequentially each phase consists of a set of commands sample convention: Of the rules in a
compound command, select the one that applies to the longest suffix.
Full morphologial analysis modest benefit !!
sses ss, ies i, ational ate, tional tion
Thesauri
Handle synonyms and homonyms Hand-constructed equivalence classes
e.g., car = automobile e.g., macchina = automobile = spider
List of words important for a given domain
For each word it specifies a list of correlated words (usually,
synonyms, polysemic or phrases for complex concepts).
Co-occurrence Pattern: BT (broader term), NT (narrower
term) Vehicle (BT) Car Fiat 500 (NT)
How to use it in SE ??
Dmoz Directory
Yahoo! Directory
Information Retrieval
Statistical Properties of Documents
Statistical properties of texts
Token are not distributed uniformly They follow the so called “Zipf Law”
Few tokens are very frequent A middle sized set has medium frequency Many are rare
The first 100 tokens sum up to 50% of the text Many of these tokens are stopwords
K-th most frequent term has frequency approximately 1/k; or the product of the frequency (f) of a token and its rank (r) is almost a constant
The Zipf Law, in detail
f = c |T| / r
r * f = c |T|f = c |T| / r
General Law
Sum after the k-th element is ≤ fkk/(z-1)
For the initial top-elements is a constant
An example of “Zipf curve”
Zipf’s law log-log plot
Consequences of Zipf Law
Do exist many not frequent tokens that do not
discriminate. These are the so called “stop words” English: to, from, on, and, the, ... Italian: a, per, il, in, un,…
Do exist many tokens that occur once in a text and thus are poor to discriminate (error?).
English: Calpurnia Italian: Precipitevolissimevolmente (o, paklo)
Words with medium frequency Words that discriminate
Other statistical properties of texts
The number of distinct tokens grows as The so called “Heaps Law” (|T|where ) Hence the token length is (log |T|)
Interesting words are the ones with Medium frequency (Luhn)
Frequency vs. Term significance (Luhn)
Related Documents
