The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment Thierry Declerck, Peter Wittenburg and Hamish Cunningham DFKI GmbH, Max-Planck-Institut für Psycholinguistik and University of Sheffield ACL/EACL2001 Workshop on Human Language Technology and Knowledge Management
20
Embed
The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment Thierry Declerck, Peter Wittenburg and Hamish Cunningham.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment
Thierry Declerck, Peter Wittenburg and Hamish Cunningham
DFKI GmbH, Max-Planck-Institut für Psycholinguistik and University of Sheffield
ACL/EACL2001 Workshop on Human Language Technology and Knowledge
Management
The MUMIS Consortium
• CTIT University of Twente, Enschede, NL NLP/IE• TSI University of Nijmegen, Nijmegen, NL ASR• DFKI Saarbrücken, D
NLP/IE• MPI Nijmegen, NL Online SW• DCS University of Sheffield, UK NLP/IE• ESTEAM Gothenburg, SE (location Athens, GR)
Translation Software
• VDA Hilversum, NL Dissemination
Objectives
• Technology development to automatically index (with formal annotations) lengthy multimedia recordings (off-line process)
Find and annotate relevant events, together with the involved entities and relations
• Technology development to exploit indexed multimedia archives (on-line process)
Search for interesting scenes and play them via Internet
Automatically transforms the speech signals into texts (for 3 languages — Dutch, English and German)
• Natural Language Processing (Information Extraction)
Analyse all available textual documents (newspapers, speech transcripts, tickers, formal texts ...), identify and extract interesting entities, relations and events
• Merging all the annotations produced so far
• Create a database with formal annotations
The Generation of Formal Annotations
• Metadata (type of game, teams, date, final score, players etc.), as they can be used a.o. for classifying and filtering videos in the MM digital archive• Events (particular actions with time codes, involved entities and related events), as they can be extracted from the video sequences• All Formal Annotations available in XML Standard
The Event TableRelated to domain ontology and multilingual terminology. Guiding the generation of formal annotations
Final whistle # 90>t>120
Subj=referee, score etc… Final score
Goal kick # 0>t>120
Subj=pl, loc=loc, cons=cons,..
Dribbling # 0>t>120
Subj=pl, loc=loc, …
Substitution # 0>t>120
Subj=pl, I.obj=pl, cause=c, …
Team (adding pl)
Red Card # 0>t>120
Subj=ref, I.obj=pl, cause=c, …
Team (red at t)
Goal # 0>t>=pen
Subj=pl, I.obj=team, score=s,
Order of goal
…
Event ID
Time Subcat/Modification Metadata
Off-line Task
Events indexed in video recording
1:0
60 m25 m25 m
SchollBasler
CampbellMatthäusBaslerNeville
DribblingFreekickFoul
28min24 min18 min17 min
DefensePassGoalFreekick
Radio Commenting3 Languages
Radio Commenting3 Languages
Radio Commenting3 Languages
Audio Commenting (TV, Radio)3 Languages
NewspaperText
NewspaperText
NewspaperText
NewspaperTexts
3 Languages
NewspaperText
NewspaperText
NewspaperText
Close caption3 Languages
multilingual IE
=> event tables
Merging of Annotations
Event = goal Player = Basler
Dist. = 25 m Time = 18
Score = 1:0
Event = goal Type = Freekick Player = Basler Dist. = 25 m Time = 17
Player = Basler Team = GermanyTime = 18 Score = 1:0 Final score = 1:0 Distance = 25 m
The Role of IE in MUMIS
• Information Extraction (IE) is the task of identifying, collecting and normalizing relevant information for a specific application or user.
• The relevant information is typically represented in form of predefined “templates”, which are filled by means of Natural Language (NL) analysis (Template = Event Table in MUMIS)
• IE combines pattern matching mechanisms, (shallow) NLP and domain knowledge (terminology and ontology).
Extension of our IE system in MUMIS
• Multilingual and multisource IE. Incremental information building
• Cross-document co-reference resolution• Combine Metadata and event extraction =>
better organisation and dynamic updating of information (KM)
• Multiple presentation of results: Template, Event table and Hyperlinks (Named Entities, rel. to KM)
Example of Processing Formal Texts
• Formal Text
• The Formal Text annotated with domain-specific information
Example of Processing Semi-Formal Texts
• Semi-Formal Text
• The Semi-Formal Text annotated with domain-specific information
Merging Component
• Acting on the generated formal annotations (Metadata and Events), but also interleaving with the generation process of those
• Checking consistency, eliminating redundancy (Template Merging), in accordance with domain ontology
• Completing the information with domain knowledge, inference Machine
On-line Tasks
Searching and Displaying
• Search for interesting events with formal queriesGive me all goals from Overmars shot with his head in 1.
• high storage capacity (n TB, 1 h MPEG1 = 1 GB)• powerful media servers / powerful network
RAID
TapeLibrary
FC Switch
Media Server
Media Server
GB Switch
Internet
1GbpsGb-Switch
Router
UI / Annotation
• UI optimization• thumbnails not that informative• which thumbnail? (several around time mark)• automatic thumbnail adjustment?• seamless operation for user• lexicon/ontology guidance• user driven input
• Manual annotation tools• MediaTagger• EUDICO
Gain - User Group
• What gets lost? Is it necessary?• Potential: direct Internet Service, less
dependencies
Current Procedure MUMIS Procedure
Manual Video Annotation Automatic Video Annotation and DB