Semantic Matching using Neural Networks

Semantic Matching using Neural Networks

Information Retrieval

CSE, IIT Kharagpur

March 20th, 2020

Information Retrieval (IIT Kharagpur) Semantic Matching using Neural Networks March 20th, 2020 1 / 21

Semantic Matching

Definition“.. conduct query/document analysis to represent the meanings ofquery/document with richer representations and then perform matching withthe representations.”

i.e., go beyong keyword (lexical) matching.We will discuss both unsupervised and supervised methods of semanticmatching.


- -

idt LM dry

-

(pseudoT-

Relevance Beyond keyword maths?Feedback → Query Expansion-

- - -

Semantic Matching: What have we seen till now?

Query expansion

Relevance Feedback

Translation Model (How to model word similarity?)

Disrtributional HypothesisWords that occur in similar contexts tend to have similar meanings.

Word embeddings have proved to be very important for modeling semanticsimilarity


-=-3


Query expansion

Relevance Feedback





-

word similarity

DitMdels

doing teary flaky

±.

a.Biasi:L :

Pointwise Mutual information


Query expansion

Relevance Feedback





= --

Word2Vec – A distributed representation

Distributional representation – word embedding?Any word wi in the corpus is given a distributional representation by anembedding

wi 2 Rd

i.e., a d�dimensional vector, which is mostly learnt!


-

\

without anyannotations

Word2Vec – A distributed representation

Distributional representation – word embedding?Any word wi in the corpus is given a distributional representation by anembedding

wi 2 Rd

i.e., a d�dimensional vector, which is mostly learnt!


→ 8 -dim

Two Variations: CBOW and Skip-grams


- d

-d

- -

What do we finally have?

For each word wi in vocabulary (size V), we have two vectors: vINi and

vOUTi , each of d�dimensions.

Generally, you can just add these vectors and use vi = vINi + vOUT

i

Ideally, similar words will have similar vectors

How do we go about using these for the retrieval task


- -

-

--

- -

-

cosine simty

Query Expansion

Pre-trained word embeddings for query expansion

Basic IdeaIdentify expansion terms using word2Vec cosine similaity

Pre-retrieval: Taking nearest neighbors of query terms as the expansionterms

Post-retrieval: Using a set of pseudo-relevant documents to restrict thesearch domain for the candidate expansion terms.


--

I#inters.ee#q-tfuisdmt-TfEtops-

Neural Translation Language Model

Language Model: Using Query LikelihoodP(q|d) = ’tq2q p(tq|d)

What happens in translation language model

p(tq|d) = Âtd2d p(tq|td)p(td|d)

You can use similarity between term embeddings for term-term translationprobability, thus


(type mismatch

q . songs d : hits←

Neural Translation Language Model

Language Model: Using Query LikelihoodP(q|d) = ’tq2q p(tq|d)

What happens in translation language modelp(tq|d) = Âtd2d p(tq|td)p(td|d)

You can use similarity between term embeddings for term-term translationprobability, thus


✓

-#I⇐ Petey

Dual Embedding Space Model (DESM)

Nalisnick et al., 2016. Improving Document Ranking with Dual WordEmbeddings. (WWW ’16 Companion).


Ss

DualEmbeddedSogndal

-

win . vous#

gimenez IEEE -

Dual Embedding Space Model (DESM)

IN-IN and OUT-OUT cosine similarities are high for words that are similarby function or type (typical) and the

IN-OUT cosine similarities are high between words that often co-occur inthe same query or document (topical).


←BTP J ER system gin d.out

O_0dn

I I

=L universities /Names-

-

Pre-trained word embeddings for document retrieval


a. ① -

-

I an

OUT- OUT IN- for

How do you evaluate this?


→-_ 7.5K¥→I-

: Top 15 doc.-

- --- returned by

- I 12 system-

② ff_idfu-

-- - tf~

Vabsets

Results: Reranking k-best list


→-t

-

- ⇒

Results: whole ranking system


- - -

→

-

⇐IUnsupervised

Semantic Matching – with Supervision


learn how ←9 dd! Freeto represent &a

Tri -

-

-I-

-

I a -

=- =

=-

DSSM

Why supervised?We learn to represent queries and documents in the latent vector space byforcing the vector representations

for relevant query-document pairs (q,d+) to be close in the latent space;and

for irrelevant query-document pairs (q,d�) to be far in the latent vectorspace


-

--

=-

Understanding DSSM - How to represent text


5%

t - - gook dim

>I

31

Bat mammal

-1 Wood

2 Necks

Understanding DSSM - Architecture


>1dg -- 3

4 30 I 2

# ok

-

%e5.gg .

.

.

! me ...

*⇒ disease:we

marmotin:÷÷÷÷÷.@does as gas

its

maxilla- I-th' % .

edao%~

DSSM - Training Objective


Evaluation Details

16,510 English queries sampled from one year query log files of Bing

Each query is associated with 15 web document titles

Relevance judgement on a scale of 0 to 4


-

-

DSSM - Results


u

① DEMI. . oursin

wisedword {

IntfRoofsupervised 2

② flauntyq D⇒ I

9 ②Seg . of words

grainMerge wqd↳,

( O words

soon.#t¥#÷÷¥RAIN

LSTM

Semantic Matching using Neural Networks

Documents