Acknowledge 07 Automated Retrieval And Categorization Of Texts In An E Learning Environment The Riks Demonstrator

1

Automated Information Retrieval and Text Categorization: The RIKS Demonstrator

Acknowledge final eventNovember 25, 2008

Marie-Francine Moens, Erik Boiy, Javier Arias (HMDB-LIIR)Saskia Debergh (i.Know)

Philippe De Lombaerde, Birger Fühne (UNU-CRIS)

Overview

UNU CRIS: The RIKS Demonstrator• UNU-CRIS: The RIKS Demonstrator• K.U.Leuven:

– Content extraction from multilingual Web pages– Text categorization: machine learning approach– Search engine and indexing infrastructure– Interfacing the Acknowledge platform

Acknowledge 25-11-2008

Interfacing the Acknowledge platform • i.Know:

– Information forensics

2

The RIKS Demonstrator• United Nations University – Comparative RegionalUnited Nations University Comparative Regional

Integration Studies (UNU-CRIS)• Issues addressed in research and capacity building:

– (i) emergence of regional (= supra-national) governance level

– (ii) linkages with other governance levels (national, global/UN)


– (iii) building of regional institutions– (iv) growing regional interdependence, etc.

• RIKS = Regional Integration Knowledge System(UNU-CRIS and GARNET NoE)


3

The RIKS DemonstratorIssues addressed in the demonstrator:

How to automate retrieval and processing p g(cleaning, search, categorization, presentation) of particular types of relevant information in an e-learning environment?:– ‘News’: short texts, various formats, dynamic

collection, short life cycle, role of news in e-learning application


– ‘Documentation’: heterogeneous texts: scientific articles, theses, essays, ... , rather static collection

– Treaty texts: long and complex texts, static collection, issue of accessibility

RIKSexample output


4

Demo


K.U.Leuven: Content extraction from multilingual Web pages

• = Extracting main content from Web page and removing extraneous data (navigation menu’s, advertisements, etc.)

• Requirements of the tool:– Accurate

Generic


– Generic– Multilingual– Fast

5



[Arias et al. submitted]

6


[Arias et al. submitted][5] =[Gottron 2008]

K.U.Leuven:Text categorization• Heterogeneous documentation and Google News

classified into 27 categories (e g trade poverty )classified into 27 categories (e.g., trade, poverty, ...)• Supervised classifier: Multinomial Naïve Bayes, Support

Vector Machine, ...• Features:

– different features: unigrams, bigrams, feature item sets, ...

• Additional feature Selection: – Chi Square, Information Gain, Linear Classifier

Weights, Orthogonal Centroid Feature Selection• Different test set ups

7

K.U.Leuven: Text categorization


RIKSK.U.Leuven: search engine


8


Demo


9

Weten dat je niet weet wat je zou moeten wetenWeten dat je niet weet wat je zou moeten weten

1. Information Forensics ‐ Smart Indexingmore than just an index

distinguishes between concepts and relationsdistinguishes between concepts and relations

starts from unstructured text (bottom‐up instead of top‐down)

Top‐down: Bottom‐up:

recognises word groups as meaningful units

Acknowledge 25-11-2008© i.Know NV ‐ All rights reserved.

knowledgeknowledgekeywords

textconcepts and relations

text


1. Information Forensics – Smart Indexing

De Fortis Bank werd overgenomen door BNP Paribas.

Traditional indexing (keywords):


stopwords

stemming

calculation

correlation

Bank

werd

Keyword Index

Fortis

0.38

0.08

0.23


stemming correlation

De Fortis Bank werd overgenomen door BNP Paribas

overgenomen

door

BNP

Paribas

0.21

0.12

0.34

0.27

10


1. Information Forensics – Smart Indexing


Smart Indexing (concepts and relations):


relation detection

concept detection

Smart Index

Concept

werd overgenomen door

Fortis Bank

Relation


De Fortis Bank werd overgenomen door BNP Paribas

werd overgenomen doorRelation

Concept BNP Paribas


2. Categorisation based on Smart Indexing

Preconditions:

Pre‐defined taxonomy/ontologyPre defined taxonomy/ontology

Top‐down processing

Advantages of Smart Indexing:

Smart Indexing Results can be used to fill and enrich the taxonomy, thus ensuring the entries are

relevant


relevant

precise

complete

11


2. Categorisation

Categorisation

Smart Indexing (concepts and relations):

The Agreement will be applied with European and withthe EFTA states.the

EFTAEU


The Agreement will be applied with the European Union and with the EFTA states.

Union

Input:

RIKSi.Know: news categorization


12

RIKSi.Know: news categorization



13


Demo


14

Thank you


Acknowledge 07 Automated Retrieval And Categorization Of Texts In An E Learning Environment The Riks Demonstrator

Documents