Latent Semantic Indexing For Information Retrieval

Latent Semantic Indexing

Sudarsun. S., M.TechChecktronix India Pvt Ltd,Chennai [email protected]

Sudarsun - Checktronix R & D 2

What is NLP ?

What is Natural Language ? Can a machine understand NL ?How are we understanding NL ?How can we make a machine understand NL ?What are the limitations ?


Major Entities …

What is Syntactic Analysis ? Deal Synonymy Deal Polysemy ?

What is Semantics ? Represent meanings as a Semantic Net

What is Knowledge ? How to represent knowledge ?

What are Inferences and Reasoning ? How to use the accumulated knowledge ?


LSA for Information Retrieval

What is LSA?Singular Value DecompositionMethod of LSAComputation of Similarity using CosineMeasuring SimilaritiesConstruction of Pseudo-documentLimitations of LSAAlternatives to LSA


What is LSA

A Statistical Method that provides a way to describe the underlying structure of texts

Used in author recognition, search engines, detecting plagiarism, and comparing texts for similarities

The contexts in which a certain word exists or does not exist determine the similarity of the documents

Closely models human learning, especially the manner in which people learn a language and acquire a vocabulary


Multivariate Data Reduction technique.

Reduces large dataset to a concentrated dataset containing only the significant information from the original data.

Singular Value Decomposition


Mathematical Background of SVD

SVD decomposes a matrix as a product of 3 matrices.

Let A be matrix of m x n, then SVD of A is

SVD(A) = UMxKSKxKVtKxN

U, V Left and Right Singular matrices respectively

U and V are Orthogonal matrix whose vectors are of unit length

S Diagonal matrix whose diagonal elements are Singular Values arranged in descending order

K Rank of A; K<=min(M,N).


Computation of SVD

To Find U,S and V matrices

Find Eigen Values and their corresponding Eigen Vectors of the matrix AAt

Singular values = Square root of Eigen Values.

These Singular values arranged in descending order forms the diagonal elements of the diagonal matrix S.

Divide each Eigen vector by its length.

These Eigen vectors forms the columns of the matrix U.

Similarly Eigen Vectors of the matrix AtA forms the columns of matrix V.

[Note: Eigen Values of AAt and AtA are equal.]


Eigen Value & Vectors

A scalar Lamba is called an Eigen Value of a matrix A if there is a non-zero vector V such that A.V = Lamba.V. This non-zero vector is the Eigen vector of A.Eigen values can be found by solving the equation | A – Lamba.I | = 0.


How to Build LSA ?

Preprocess the document collection Stemming Stop words removal

Build Frequency MatrixApply Pre-weightsDecompose FM into U, S, VProject Queries


Step #1: Construct the term-document matrix; TDM One column for each document One row for every word The value of cell (i, j) is the frequency of word i in document j

Frequency Matrix




Step #2: Weight FunctionsIncrease the efficiency of the information retrieval.Allocates weights to the terms based on their occurrences.

Each element is replaced with the product of a Local Weight Function(LWF) and a Global Weight Function(GWF).

LWF considers the frequency of a word within a particular text

GWF examines a term’s frequency across all the documents.

Pre-weightingsApplied on the TDM before computing SVD.

Post-weightingsApplied to terms of a query when projected for matching or searching.


Step #3: SVD

Perform SVD on term-document matrix X.

SVD removes noise or infrequent words that do not help to classify a document.

Octave/Mat lab can be used

[u, s, v] = svd(A);


A U

S

Vt

m x n m x k k x k k x n

· ·

Ter

ms

Documents

0

0


Documents

TDM

SVD

Terms

U S V


Similarity Computation Using Cosine

Consider 2 vectors A & B. Similarity between these 2 vectors is

A.B CosØ = ------------------

|A|. |B|

CosØ ranges between –1 to +1


Similarity Computations in LSA


Term-term SimilarityCompute the Cosine for the row vectors

of term ‘i’ and term ‘j’ in the U*S matrix.

US


Document – Document Similarity

Compute the Cosine for the column vectors of document ‘i’ and document ‘j’ in the S*Vt matrix.SVt


Term – Document Similarity

Compute Cosine between row vector of term ‘i’ in U*S1/2 matrix and column vector of document ‘j’ in S1/2*Vt matrix.


U*S1/2

S1/2*Vt


Construction of Pseudo-document A Query is broken in to terms and

represented as a column vector (say ‘q’) consisting of ‘M’ terms as rows.

Then Pseudo-document (Q) for the query(q) can be constructed with the help of following mathematical formula.

Q = qt*U*S-1

After constructing the Pseudo-document, we can compute the similarities of query-term, query-document using earlier mentioned techniques.


Alternatives to LSA

LSA is limited to Synonymy problem

PLSA – Probabilistic Latent Semantic Analysis to handle Polysemy.

LDA – Latent Dirichlet Allocation.


References

http://www.cs.utk.edu/~lsi/papers/http://www.cs.utk.edu/~berry/lsi++http://people.csail.mit.edu/fergus/iccv2005/bagwords.htmlhttp://research.nitle.org/lsi/lsa_explanation.htmhttp://en.wikipedia.org/wiki/Latent_semantic_analysishttp://www-psych.nmsu.edu/~pfoltz/reprints/BRMIC96.htmlhttp://www.pcug.org.au/~jdowling/http://www.ucl.ac.uk/oncology/MicroCore/HTML_resource/PCA_1.htmhttp://public.lanl.gov/mewall/kluwer2002.htmlhttp://www.cs.utexas.edu/users/suvrit/work/progs/ssvd.html


Thanks..

You may send in your queries to [email protected]

Latent Semantic Indexing For Information Retrieval

Technology

singular matrices

singular values

orthogonal matrix

accumulated knowledge

unit length sdiagonal

u mxk s kxk v t kxn

concentrated dataset

significant information