Towards Methods for Efficient Access to Spoken Content in the AMI Corpus (SSCS 2010)

(gjones, meskevich, agyarmati @computing.dcu.ie) - 1 - ‏

Centre for Digital Video Processing

C e n t r e f o r D I g I t a l V I d e o P r o c e s s I n g

Towards Methods for Efficient Access to Spoken Content in the AMI Corpus

Gareth J. F. Jones Maria Eskevich Ágnes Gyarmati

Centre for Digital Video Processing School of Computing

Dublin City University, Ireland




Outline

•  Issues •  AMI corpus •  Pre-processing •  Experiment and Results •  Future work




Outline





Issues: types of Spoken Content

– News broadcast: •  Structured •  Clearly articulated speech

-> standard text document retrieval task on ASR transcript

– Other types of speech (meetings, lectures): •  Lack of clearly defined document form/structure •  Informal style, cross-talk, noisy environment

->We have to define: •  Search units •  Location of relevant items




Issues: Existing Research

•  Speech Search: –  TV and radio news: Spoken Document Retrieval

(SDR) task at TREC (2000) –  Interviews: Malach Collection (2007) –  AMI (Augmented Multi-party Interaction) corpus

•  Recognition WER and Retrieval: –  Low recognition error level:

•  little loss in retrieval effectiveness (2000) •  documents are retrieved at higher ranks (2003, 2007)

–  Specific metrics (semantic impact of substitutions): •  correlation with retrieval performance (AMI Corpus, 2009)




Issues

•  Goal: – Investigate how difference between manual

and automatic transcription accuracy influences retrieval effectiveness on the material of the AMI Corpus

•  Experiment: – Segmentation of spoken content – Known-item search task using slides from

meetings as queries




Outline





AMI Corpus

•  100 hours •  Each meetings approximately 30

minutes •  Simulating project meetings •  4-5 participants •  Headset and circular microphones •  Automatic and manual transcripts

available •  Additional data (slides, minutes)




Outline





Pre-processing: segmentation

•  Linear segmentation (C99 algorithm): Cosine based sequential sentence similarity

based algorithm Boundaries inserted between sentences based on the difference of lexical inventory (stemmed)

•  Time segmentation (approximately 90 seconds)




Pre-processing: segmentation

•  Number of segments

•  Average number of words per segment

Type of transcript Linear segmentation (C99)

Manual transcript 2678

ASR transcript 3831

Type of transcript Linear segmentation (C99)

Manual transcript 320 ASR transcript 221




Pre-processing: Word Recognition Rate (WRR)

1.  Alignment between ASR and manual transcripts

2.  Recognition rate count Recognition rate – number of correctly recognized words in the meeting divided by the total number of words in the transcript

3.  Recognition rate without stop words




Relation between segmentation and recognition rate




Pre-processing: cross-segmentation




Outline





Experiment: slides and relevant segments selection




Experiment: slides and relevant segments selection

Type of queries

Number of

queries

Number of relevant segments with segmentation based on

ASR transcript Manual transcript

Min 15 56 49

Max 24 68 39

Random 25 36 42




Experiment: Indexing & Retrieval Setup

•  Indri language model of the open source Lemur Toolkit (http://www.lemurproject.org/): –  texts are stemmed using Lemur's built-in

Porter stemmer

•  Stopword list provided by Snowball (http://snowball.tartarus.org/)




Results: at ranks 100 •  Recall at ranks 100:

•  Mean Reciprocal Rate at ranks 100:




Outline

•  Issues •  AMI corpus •  Pre-processing •  Experiment •  Results •  Future work




Problems

•  Errors in the ASR output

•  Common knowledge of the participants of the meeting -> some words are not spoken

•  All parts of the meetings are indexed in the same way

•  Retrieval algorithm favours longer segments




Future work

•  Construct proper segment-based relevance set for the slides

•  Analysis of ASR errors influence on segmentation

•  ASR transcript improvement




Thank You Thank you for your attention!

Questions?

Towards Methods for Efficient Access to Spoken Content in the AMI Corpus (SSCS 2010)

Technology