Unsupervised Does Not Mean Uninterpretable: The Case for Word Sense Induction and Disambiguation

Universität Mannheim – Name of Presenter: Titel of Talk/Slideset (Version: 27.6.2014) – Slide 1

Unsupervised Does Not Mean Uninterpretable: The Case for Word Sense Induction and

Disambiguation

Chris [email protected]

Alexander [email protected]

Stefano [email protected]

Simone Paolo [email protected]

Eugen [email protected]

Universität Mannheim – Stefano Faralli: Unsupervised Doesn’t Mean Uninterpretable (05.04.2017) – Slide 2

Word Sense Disambiguation and Word Sense Induction

- Word Sense Disambiguation (WSD) is the ability to computationally determine which sense of a word is activated by its use in a particular context (Navigli, 2009).

- Word Sense Induction (WSI) concerns the automatic identification of the senses of a word (Agirre and Soroa, 2007).


Motivation

- Knowledge-based sense representations:

- Definitions, synonyms, taxonomic relations and images make this representation easily interpretable.

BabelNet.org

http://babelnet.org

http://babelnet.org


Motivation

- Unsupervised knowledge-free sense representations:

- Absence of definitions, hypernyms, images, and the dense feature representation make this representation uninterpretable.

- RQ: Can we make unsupervised knowledge-free sense representations interpretable?

...

AdaGram sense embeddings (Bartunov et al., 2016)


SOTA

Supervised approaches (Ng, 1997; Lee and Ng, 2002; Klein et al., 2002; Wee, 2010; Zhong and Ng, 2010):

- require large amounts of sense-labeled examples per target word


SOTA

Supervised approaches (Ng, 1997; Lee and Ng, 2002; Klein et al., 2002; Wee, 2010; Zhong and Ng, 2010):

- require large amounts of sense-labeled examples per target word

Knowledge-based approaches:

- “Classic” approaches (Lesk,1986; Banerjee and Pedersen, 2002; Pedersen et al., 2005; Miller et al., 2012; Moro et al., 2014):

- Approaches based on sense embeddings (Chen et al., 2014; Rothe and Schütze, 2015; Camacho-Collados et al., 2015; Iacobacci et al., 2015; Nieto Pina and Johansson, 2016)

- require manually created lexical semantic resources


SOTA

Unsupervised knowledge-free approaches:

- Context clustering (Pedersen and Bruce, 1997; Schütze, 1998) including knowledge-free sense embeddings (Huang et al., 2012; Tian et al., 2014; Neelakantan et al., 2014; Li and Jurafsky, 2015; Bartunov et al., 2016)

- Dense vector sense representations are not interpretable.


SOTA

Unsupervised knowledge-free approaches:

- Context clustering (Pedersen and Bruce, 1997; Schütze, 1998) including knowledge-free sense embeddings (Huang et al., 2012; Tian et al., 2014; Neelakantan et al., 2014; Li and Jurafsky, 2015; Bartunov et al., 2016)

- Dense vector sense representations are not interpretable.

- Word ego-network clustering methods (Lin, 1998; Pantel and Lin, 2002; Widdows and Dorow, 2002; Biemann, 2006; Hope and Keller, 2013):

- Sparse interpretable graph representations: used in our work.


Unsupervised & Knowledge-Free & Interpretable WSD

A novel approach to WSD/WSI:✓ Unsupervised &✓ Knowledge-free &✓ Interpretable



A novel approach to WSD/WSI:✓ Unsupervised &✓ Knowledge-free &✓ Interpretable

Outline of our method for word sense induction and disambiguation:



Context Features:

- Dependency Features

- Co-occurrence Features

- Language Model Features (3-grams):



Word and Feature Similarity Graphs:

- Count-based approach, JoBimText (Biemann and Riedl, 2013)- Dependency-based features- LMI normalization (Evert, 2005)- 1000 most salient features- 200 most similar words per term

+ Sparse interpretable features + Performance is comparable to word2vec/f (Riedl, 2016)



Word Sense Induction: Ego-Network Clustering (Biemann, 2006):



Labeling Induced Senses with Hypernyms and Images:



Word Sense Disambiguation with Induced Word Sense Inventory:

- Max. similarity of context and sense representations.



Word Sense Disambiguation with Induced Word Sense Inventory:

- Max. similarity of context and sense representations.



Interpretability of the model at three levels:

(1) Word Sense Inventory (2) Sense Feature Representations

(3) Disambiguation in Context













- Hypernymy labels help to interpret the automatically induced senses.

- A live demo: https://goo.gl/eXgRg4

The best match

The context

The worst match

http://goo.gl/eXgRg4

http://goo.gl/eXgRg4



Similarity of the context and the sense representation based on the sparse features can be traced back:

Features that explain the sense choice


Evaluation

RQ 1:

Which combination of unsupervised features yields the best results?


Evaluation

RQ 1:


RQ 2:

How does the granularity of an induced inventory impact performance?


Evaluation

RQ 1:


RQ 2:


RQ 3:

What is the quality of our approach compares to SOTA unsupervised WSD systems, including those based on the uninterpretable models?


RQ1: Dataset and Evaluation Metrics


- Dataset:- TWSI 2.0: Turk Bootstrap Word Sense Inventory (Biemann, 2012) - 2,333 senses (avg. polysemy of 2.31)- 145,140 annotated sentences: the full version- 6,166 annotated sentences: the sense-balanced version

- No monosemous words




- Dataset:- TWSI 2.0: Turk Bootstrap Word Sense Inventory (Biemann, 2012) - 2,333 senses (avg. polysemy of 2.31)- 145,140 annotated sentences: the full version- 6,166 annotated sentences: the sense-balanced version

- No monosemous words

- Evaluation Metrics:- Mapping the induced inventory to the gold sense inventory - Precision, Recall, F1


RQ1: Results


- Baselines: MFS, Random, LCB,...


RQ1: Results


- Performance of various features and their combinations:------------ Full and the sense-balanced TWSI datasets based on the coarse

inventory with 1.96 senses/word (N = 200, n = 200).


RQ2: Results


- Full and the sense-balanced TWSI datasets- Wikipedia corpus - The coarse inventory (1.96 senses per word)


RQ2: Results




RQ2: Results






Dataset: - SemEval 2013 Task 13: WSI for Graded and Non-Graded Senses- 4,664 contexts- 6,73 senses per word




Dataset: - SemEval 2013 Task 13: WSI for Graded and Non-Graded Senses- 4,664 contexts- 6,73 senses per word

Evaluation Metrics:

- Supervised metrics (Jacc. Ind., Tau, WNDCG)- Requite mapping of the induced sense inventory to the gold inventory

- Unsupervised metrics (Fuzzy NMI, Fuzzy B-Cubed)- No mapping is required



State-of-the-art methods:- AI-KU, Unimeld, UoS, La Sapienze -- SemEval participants- AdaGram and SenseGram -- sense embeddings-


Conclusions

− We presented a novel approach to WSD:• Unsupervised +• Knowledge-free +• Based on ego-network clustering +• Interpretable at the levels of

i. Word sense inventoryii. Sense representationsiii. Disambiguation results

− The method yields SOTA results, comparable to uninterpretable models, e.g. sense embeddings, while being human-readable.

− Interpretability of the knowledge-based sense representations, can be achieved using unsupervised knowledge-free framework.

goo.gl/eXgRg4


Want to see a demo? https://goo.gl/eXgRg4

https://goo.gl/eXgRg4

Universität Mannheim – Name of Presenter: Titel of Talk/Slideset (Version: 27.6.2014) – Slide 37

We acknowledge the support of:

Development of the demo: Fide Marten


References

- Eneko Agirre and Philip G. Edmonds. 2007. Word sense disambiguation: Algorithms and applications, volume 33. Springer Science & Business Media.- Satanjeev Banerjee and Ted Pedersen. 2002. An adapted Lesk algorithm for word sense disambiguation using WordNet. In Proceedings of the Third International

Conference on Intelligent Text Processing and Computational Linguistics, pages 136–145, Mexico City, Mexico. Springer.- Sergey Bartunov, Dmitry Kondrashkin, Anton Osokin, and Dmitry Vetrov. 2016. Breaking sticks and ambiguities with adaptive skip-gram. In Proceedings of the 19th

International Conference on Artificial Intelligence and Statistics (AISTATS’2016), Cadiz, Spain.- Osman Baskaya, Enis Sert, Volkan Cirik, and Deniz Yuret. 2013. AI-KU: Using Substitute Vectors and Co-Occurrence Modeling for Word Sense Induction and

Disambiguation. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), volume 2, pages 300–306, Atlanta, GA, USA. Association for Computational Linguistics.

- Chris Biemann and Martin Riedl. 2013. Text: Now in 2D! a framework for lexical expansion with contextual similarity. Journal of Language Modelling, 1(1):55–95.- Chris Biemann. 2006. Chinese Whispers: an efficient graph clustering algorithm and its application to natural language processing problems. In Proceedings of the

first workshop on graph based methods for natural language processing, pages 73–80, New York City, NY, USA. Association for Computational Linguistics.- Chris Biemann. 2012. Turk Bootstrap Word Sense Inventory 2.0: A Large-Scale Resource for Lexical Substitution. In Proceedings of the 8th International Conference

on Language Resources and Evaluation, pages 4038–4042, Istanbul, Turkey. European Language Resources Association.- Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2015. SensEmbed: learning sense embeddings for word and relational similarity. In Proceedings of

the 53rd Annual Meeting of the Association for Computational Linguistics (ACL’2015), pages 95–105, Beijing, China. Association for Computational Linguistics.- Nancy Ide and Jean Veronis. 1998. Introduction to ´ the special issue on word sense disambiguation: the state of the art. Computational linguistics, 24(1):2– 40.- David Jurgens and Ioannis Klapaftis. 2013. Semeval- 2013 Task 13: Word Sense Induction for Graded and Non-graded Senses. In Proceedings of the 2nd Joint

Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval’2013), pages 290–299, Montreal, Canada. Association for Computational Linguistics.

- Jey Han Lau, Paul Cook, and Timothy Baldwin. 2013. unimelb: Topic Modelling-based Word Sense Induction. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), volume 2, pages 307–311, Atlanta, GA, USA. Association for Computational Linguistics.

- Michael Lesk. 1986. Automatic Sense Disambiguation Using Machine Readable Dictionaries: How to Tell a Pine Cone from an Ice Cream Cone. In Proceedings of the 5th annual international conference on Systems documentation, pages 24–26, Toronto, ON, Canada. ACM.

- Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity linking meets word sense disambiguation: a unified approach. Transactions of the Association for Computational Linguistics, 2:231– 244.

- Roberto Navigli and Simone Paolo Ponzetto. 2010. Babelnet: Building a very large multilingual semantic network. In Proceedings of the 48th Annual Meeting of the Association of Computational Linguistics, pages 216–225, Uppsala, Sweden.

- Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient non-parametric estimation of multiple embeddings per word in vector space. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1059–1069, Doha, Qatar. Association for Computational Linguistics.

- Luis Nieto Pina and Richard Johansson. 2016. Em- ˜ bedding senses for efficient graph-based word sense disambiguation. In Proceedings of TextGraphs-10: the Workshop on Graph-based Methods for Natural Language Processing, pages 1–5, San Diego, CA, USA. Association for Computational Linguistics

- Martin Riedl. 2016. Unsupervised Methods for Learning and Using Semantics of Natural Language. Ph.D. thesis, Technische Universitat Darm- ¨ stadt, Darmstadt.- Sascha Rothe and Hinrich Schutze. 2015. Autoex- ¨ tend: Extending word embeddings to embeddings for synsets and lexemes. In Proceedings of the 53rd Annual

Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1793–1803, Beijing, China. Association for Computational Linguistics

- Jean Veronis. 2004. HyperLex: Lexical cartogra- ´ phy for information retrieval. Computer Speech and Language, 18:223–252.


RQ4: Results

Does feature expansion improves performance?


Full TWSI Sense-Balanced TWSI


RQ4: Results





RQ4: Results


