C@MERATA at MediaEval 2014 - Extracting Answer Passages from Classical Music Scores using Natural Language Descriptions Richard Sutcliffe, University of Essex Tim Crawford, Goldsmiths, University of London Chris Fox, University of Essex Eduard Hovy, Carnegie-Mellon University Deane Root, University of Pittsburgh
77
Embed
Camerata at MediaEval 2014 - Extracting Answer Passages from Classical Music Scores using Natural Language Descriptions
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
C@MERATA at MediaEval 2014 - Extracting Answer Passages from
Classical Music Scores using Natural Language Descriptions
Richard Sutcliffe, University of Essex Tim Crawford, Goldsmiths, University of London
Chris Fox, University of Essex Eduard Hovy, Carnegie-Mellon University
Deane Root, University of Pittsburgh
2
Outline Background
Task Design
Passages and Divisions
Query Types and Music Scores
Gold Standard
Evaluation Metrics
XML Formats
Baseline System
Campaign this Year
Results
Conclusions
3
C@merata stands for: Cl@ssical Music Extraction of Relevant Aspects by Text Analysis
4
C@merata Task There is a series of questions with required answers. Provided Question:
• A short noun phrase in English referring to musical features in a score,
• A short classical music score in MusicXML. Required Answer:
• The location(s) in the score of the requested musical feature. We call each such location a passage.
5
Motivation When studying musicological analyses of works of western classical art music there are frequent references to relevant passages in the printed score. Our work if successful could enable such references to be associated with passages automatically, hence facilitating the scholarly work of musicologists and musicians, both professional and amateur. It could also enable passages to be found interactively on the basis of an input natural language phrase.
6
Organisers Richard Sutcliffe, University of Essex (has worked on CLEF QA evaluations for ten years). Tim Crawford, Goldsmiths, University of London (lutenist with the Parley of Instruments, musicologist specialising in electronic editions of works by Henry Purcell, Silvius Leopold Weiss etc). Eduard Hovy, Carnegie-Mellon University (has worked extensively on NLP and QA at TREC, MUC, TIPSTER, CLEF, Machine Reading etc). Deane Root, University of Pittsburgh (Editor-in-Chief of Grove Online - a comprehensive source of information on western classical music) Chris Fox, University of Essex (has worked on NLP for many years, also has a thorough knowledge of music theory).
7
Background Originally there was Information Retrieval along with the SIGIR conference, from 1978 onwards. Interest developed in Music Information Retrieval, leading to the ISMIR conference, from 2000 onwards. TREC IR evaluations began in 1992. CLEF multi-lingual and cross-lingual evaluations began in 2000. MIREX evaluations began in 2005. They are run by the University of Illinois at Urbana-Champaign. Focus on low-level tasks like melodic similarity, mostly with audio data.
8
Background cont. Question Answering started at CLEF in 2003. In 2011, QA4MRE task started. 12 documents on 3 topics (Aids, Climate Change, Music and Society), ten MCQ questions on each. Used TED film transcripts. In 2012, QA4MRE had same topics and used various sources for the music topic, including 1911 Encyclopaedia Britanica and Wikipedia. In 2013, QA4MRE used articles from Grove Online for the music topic: ‘Johann Baptist Cramer’, ‘Electronic Dance Music’, ‘Film Music - Hollywood’, ‘Disciplines of Musicology - Analytic Traditions’.
9
Task Design There are 200 queries. We saw earlier that we have: Provided Question:
• A short noun phrase in English referring to musical features in a score,
• A short classical music score in MusicXML. Required Answer:
• The location(s) in the score of the requested musical feature. We call each such location a passage.
10
Example Query
t: pitch_and_length
q: semiquaver G#
s: corelli_allegro_tr_clementi.xml
[ 4/4, 4, 2:5-2:5 ]
[ 4/4, 4, 5:15-5:15 ]
[ 4/4, 4, 7:3-7:3 ]
...
11
Specifying Passages in a Score We specify a passage as follows:
• a time signature, • a divisions value • a start bar and beat • an end bar and beat Example:
[4/4,1,1:1-2:4] time signature is 4/4. divisions value is 1 so we count in crotchets (quarter notes) Passage starts in bar 1 before the first crotchet (i.e. 1:1) Passage ends in bar two after the fourth crotchet (i.e. 2:4) Thus passage consists of the two complete bars (measures) numbered one and two.
12
What is a Divisions Value? The number of beats into which we divide a crotchet. A suitable value depends on what we wish to demarcate. Working in whole crotchets: Divisions = 1
Working in quavers: Divisions = 2
Working in quaver triplets: Divisions = 3
Working in semiquavers or quavers or triplet quavers: Divisions = 12 !!
With Divisions = 12:
one crotchet = 12 beats
one quaver = 6 beats
one quaver triplet = 4 beats
one semiquaver = 3 beats
Note: MusicXML uses the divisions concept and we got the idea from there.
13
Query Types In C@merata there are twelve types of query:
Source of MusicXML Scores Public domain scores were obtained from sites such as musescore.com. Most of these are transcribed by amateurs. MusicXML is an interchange format. Thus, most score-writing programs e.g. Finale or Sibelius can export to MusicXML (and input from it).
16
Distribution of Scores by Number of Staves Staves Frequency
1 4 2 6 3 6 4 2 5 2
17
Example Query 1 (from earlier)
t: pitch_and_length
q: semiquaver G#
s: corelli_allegro_tr_clementi.xml
[ 4/4, 4, 2:5-2:5 ]
[ 4/4, 4, 5:15-5:15 ]
[ 4/4, 4, 7:3-7:3 ]
...
18
Example Query 2
t: followed_by
q: G sharp followed by B
s: corelli_allegro_tr_clementi.xml
[ 4/4, 4, 2:5-2:6 ]
[ 4/4, 4, 5:15-5:16 ]
[ 4/4, 4, 8:11-8:12 ]
...
19
Example Query 3
t: melodic_interval
q: rising melodic octave
s: corelli_allegro_tr_clementi.xml
[ 4/4, 4, 1:1-1:8 ]
[ 4/4, 4, 4:1-4:8 ]
[ 4/4, 4, 4:9-4:16 ]
...
20
Example Query 4
t: triad_spec
q: tonic triad
s: bach_chorale_507b.xml
[ 4/4, 1, 0:1-0:1 ]
[ 4/4, 1, 7:1-7-1 ]
21
Creation of the Gold Standard of 200 Questions Stage 1
20 Baroque scores were collected. 4 on 1 stave, 6 on 2 staves etc. 10 questions set on each score. 10 scores: English terminology, 10 scores: American terminology. Mix of question types set against each score. For each question, answers were found in the score. Gradually built up the required question-type distribution. Questions were recorded in C@merata ascii format.
22
Creation of the Gold Standard (cont.) Stage 2 Answers from Stage 1 were carefully checked. (Precision was good, Recall was sometimes lacking.) Stage 3
Questions in C@merata ascii format automatically converted into C@merata XML format. Stage 4
Test questions automatically extracted from Gold Standard (minus the answers).
23
Evaluation of a Returned Passage Passage is beat correct if it starts at exactly the correct beat (as specified by the divisions value) in the correct start bar and also ends at the correct beat in the end bar. - Useful for applications of results which are themselves automatic. Passage is measure correct if it starts in the bar where the requested feature starts and ends in the bar where the requested feature ends. - Sufficient in many cases for humans to use.
24
Example Query 1: Beat correct vs. Measure Correct
t: pitch_and_length
q: semiquaver G#
s: corelli_allegro_tr_clementi.xml
[ 4/4, 4, 2:5-2:5 ]
[ 4/4, 4, 5:15-5:15 ]
[ 4/4, 4, 7:3-7:3 ]
...
25
Beat Precision (BP) is the number of beat correct passages returned by a system divided by the number of passages (correct or incorrect) returned. Beat Recall (BR) is the number of beat correct passages returned by a system divided by the total number of answer passages known to exist. Measure Precision (MP) is the number of measure correct passages returned by a system divided by the number of passages (correct or incorrect) returned. Measure Recall (MR) is the number of measure correct passages returned by a system divided by the total number of answer passages known to exist.
26
XML Question Format <questions task="camerata" year="2014" task_no="1"
Baseline System To encourage participation we created a baseline C@merata system Written in Python 27 Uses excellent Music21 of Michael Scott Cuthbert (Will convert MusicXML into a tractable form and manipulate it) Handles all the basic input of questions, processing and output Shows how to use Music21 Actual question-answering component is skeletal
29
Architecture of Baseline System 1. Analyse the input xml file and extract questions and score files 2. For each question:
• Parse question using Stanford Parser • Determine question type (similar to normal QA) • Parse the score file using Music21 • Based on question type, search for answer passages in score 3. Write out answers to output xml file
30
Campaign this Year Fri 24.03.14 Participant registration Fri 02.05.14 Release of training data Mon 16.06.14 - Fri 20.06.14 Systems answer test questions Fri 27.06.14 Publication of results Thu 16.10.14 - Fri 17.10.14 MediaEval Workshop in Barcelona
31
Participants Runtag Leader Affiliation Country
CLAS Stephen Wan CSIRO Australia DMUN Tom Collins De Montfort University England OMDN Donncha Ó Maidín University of Limerick Ireland TCSL Nikhil Kini Tata Consultancy Services India UNLP Kartik Asooja NUI Galway Ireland
32
Results
Run BP BR BF MP MR MF
CLAS01 0.713 0.904 0.797 0.764 0.967 0.854
DMUN01 0.372 0.712 0.489 0.409 0.784 0.538
DMUN02 0.380 0.748 0.504 0.417 0.820 0.553
DMUN03 0.440 0.868 0.584 0.462 0.910 0.613
LACG01 0.135 0.101 0.116 0.188 0.142 0.162
OMDN01 0.415 0.150 0.220 0.424 0.154 0.226
TCSL01 0.633 0.821 0.715 0.652 0.845 0.736
UNLP01 0.113 0.516 0.185 0.155 0.703 0.254
UNLP02 0.290 0.512 0.370 0.393 0.692 0.501
Maximum 0.713 0.904 0.797 0.764 0.967 0.854
Minimum 0.113 0.150 0.185 0.155 0.154 0.226
Average 0.420 0.654 0.483 0.460 0.734 0.534
33
Summary of All Results Based on Beat F-score (BF) ranking is: CLAS01 (0.797), TCSL01 -(0.715), DMUN03 (0.584), UNLP02 (0.370), OMDN01 (0.220), LACG01 (0.116) All submitted runs beat the baseline Best scores were quite high, so task was perhaps quite easy
Summary of Question Type Results Ranking by question type using BF: simple_length (0.810), simple_pitch (0.677), pitch_and_length (0.644), word_spec (0.520), stave_spec (0.508), melodic_interval (0.402), perf_spec (0.339), followed_by (0.278), harmonic_interval (0.188), triad_spec (0.095), cadence_spec (0.093), texture_spec (0.072) Not all question types are equally easy First few types in the list are quite easy Last three or four types in the list are very difficult. melodic_interval easier than harmonic_interval
36
Conclusions Results are generally high So task was rather easy Some question types were easier than others Technical aspects (data formats, passages, evaluation etc) worked out well Need more complex queries for a second campaign Participants are interested in continuing
37
The End
38
39
Example Questions The following slides show some example scores and questions against them.
40
Work: J.S. Bach, Das Wohltemperierte Klavier, Book 1, Prelude No. 2 in C minor BWV 847 Extract:
41
Q: passage in common time A: [4/4,1,1:1-4:4] Q: interval of a melodic sixth A: [4/4,4,1:1-1:2], [4/4,4,1:8-1:9] [4/4,4,1:9-1:10], [4/4,4,2:1-2:2], [4/4,4,2:8-2:9], [4/4,4,2:9-2:10], [4/4,4,3:1-3:2], [4/4,4,3:8-3:9], [4/4,4,3:9-3:10] Q: second A: None, because a second is harmonic by default Q: C followed by Eb A: [4/4,4,1:5-1:6], [4/4,4,1:13-1:14], [4/4,4,4:1-4:2], [4/4,4,4:9-4:10] Q: C followed by Eb in the bass clef A: [4/4,4,4:1-4:2], [4/4,4,4:9-4:10] Q: semiquaver E natural / sixteenth note E natural A: [4/4,4,2:3-2:4]
42
Work: J.S. Bach, Suite No. 3 in C Major for Cello, BWV 1009, Sarabande Extract:
Q: harmonic interval of a minor third A: [3/4,2,208:1-208:1] Q: minor third A: [3/4,2,208:1-208:1] (thirds are harmonic by default) Q: dotted quaver / dotted eighth note A: [3/4,4,206:5-206:7], [3/4,4,207:5-207:7], [3/4,4,208:5-208:7] Q: harmonic perfect fifth A: [3/4,1,206:1-206:1], [3/4,1,207:1-207:1], [3/4,4,208:5-208:7] (in last passage, C and G should be dotted but were not in the original) Q: simultaneous harmonic perfect fifth and harmonic eleventh A: [3/4,4,208:5-208:7] (same point, C and G are assumed dotted in this example)
43
Work: J. Dowland, King of Denmark's Galliard, P 40 Extract:
Q: perfect cadence
A: [3/4,1,3:3-4:3] (We are assuming the cadence continues until the end of the bar) Q: four consecutive quavers / four consecutive quarter notes A: [3/4,2,2:3-2:6] Q: dotted minim in the bass / dotted half note in the bass A: [3/4,1,2:1-2:3] Q: harmonic fourth A: [3/4,1,1:1-1:1], [3/4,1,1:2-1:3], [3/4,1,4:1-4:3] (Note that there are two instances in bar (measure) 1 because the chord is played twice)
44
Work: J. Dowland, Pauana Dulandi, P 86 Extract:
Q: consecutive 5ths in the bass A: [4/4,1,2:1-2:4] Q: semibreve Bb in the treble clef / whole note Bb in the treble clef A: [4/4,1,1:1-1:4], [4/4,1,4:1-4:4] Q: octave followed two bars later by another octave
Work: G.F. Handel, Messiah, HG xlv, "And the Glory of the Lord" Extract:
Q: crotchet rest / quarter note rest A: [3/4,1,1:1-1:1] Q: dotted crotchet followed by three quavers / dotted quarter note followed by three eighth
notes A: [3/4,1,5:1-5:3], [3/4,1,6:1-6:3] Q: four hemidemisemiquavers / four sixty-fourth notes A: [3/4,4,3:9-3:9] Q: three quavers in a row / three eighth notes in a row A: [3/4,2,5:4-5:6], [3/4,2,6:4-6:6]
A: [4/4,1,22:1-22:1], [4/4,1,25:1-25:1] Q: D natural quaver / D natural eighth note
A: [4/4,2,23:3-23:3], [4/4,2,24:4-24:4] Q: open VII triad in the first inversion A: [4/4,1,23:3-23:3] Q: melodic fourth in the bass clef A: [4/4,1,24:1-24:2]
47
Work: D. Scarlatti, Keyboard Sonata in D minor, K 1 Extract:
48
Q: eight staccato notes in succession A: [4/4,1,17:1-17:4] Q: augmented melodic fourth A: [4/4,2,19:3-19:4], [4/4,2,20:4-20:5] Q: third
A: [4/4,4,17:13-17:13], [4/4,2,19:2-19:2], [4/4,2,19:4-19:4], [4/4,4,19:9-19:9], [4/4,4,20:9-20:9] (these are assumed to be harmonic and can thus be across parts) Q: a quaver, then a major third / an eighth note, then major third
A: [4/4,2,19:1-19:2], [4/4,2,19:3-19:4] (third assumed to be harmonic and to follow the quaver immediately) Q: change from bass to treble clef A: [4/4,1,17:2p] (recall that this means the clef change is immediately after 17:2)
49
Work: D. Scarlatti, Keyboard Sonata in D major, K 430 Extract:
A: [3/8,4,57:1-57:2], [3/8,4,57:2-57:6] (melodic must be stated, otherwise it is harmonic) Q: harmonic 5th followed by harmonic 4th A: [3/8,2,58:1-58:2], [3/8,2,62:1-62:2]
50
Architecture of Baseline System We have built a very basic system to perform the task: 1. Analyse the input xml file and extract questions and score files 2. For each question:
• Parse question using Stanford Parser • Determine question type (similar to normal QA) • Parse the score file using Music21 • Based on question type, search for answer passages in score 3. Write out answers to output xml file
51
Linguistic Observations Queries are noun phrases Head noun group is the main feature (e.g. F#) PP modifiers qualify this (e.g. in the bass clef) A lot of terminology is used (e.g. semiquaver, soprano part, forte bar) Search based on query classification will work at least for simple examples.
52
Summary We are trying to find ways to link natural language descriptions of music to musical scores. Starting with the C@merata task, we are working with very simple tasks to develop the technology. Then we will progress to more complex tasks which are of genuine interest to musicologists. In due course, we aim to tie passages in musicological texts such as in Grove Online to the corresponding music scores.
53
Music Extracts symphony opening with a horn call over shimering strings Bruckner 4th Symphony http://www.youtube.com/watch?v=J8t1TzN0RRY Start: 0:00 End: 0:44 symphony closing with six unison chords Sibelius 5th Symphony http://www.youtube.com/watch?v=nkzrSZKA4cM Start: 9:36 End: 9:56 Nielsen 4th Symphony, Finale (shows two timpanists well) http://www.youtube.com/watch?v=yXDe1hj4HBo Start: 32:10 End: 32:48
54
Extracts Not Used symphony opening with two octave descending arpeggio Beethoven 9th Symphony string quartet opening including double stopping on all four instruments Beethoven op 127 symphony featuring a battle between two timpanists Nielsen 4th Symphony, Finale (not so good) http://www.youtube.com/watch?v=sD9I-UiYfW8 Start: 6:30 End: 7:08
55
Grove’s Dictionary of Music and Musicians • 1879-1889 George Grove, civil engineer, music administrator, writer and then Director of
Royal College of Music, wrote A Dictionary of Music and Musicians in four volumes. • 1904-910 Fuller Maitland edited the second edition - Grove's Dictionary of Music and
Musicians - in five volumes. • 1927 Henry Colles edited the third edition - in five volumes. • 1940 Henry Colles edited the fourth edition - in seven volumes. • 1954 Eric Blom edited the fifth edition - in nine volumes.
56
• 1980 Stanlie Sadie edited the sixth edition - The New Grove Dictionary of Music and
Musicians - in twenty volumes. Contained 22,500 articles and 16,500 biographies. • 2001 Stanlie Sadie edited the seventh edition - in 29 volumes. This was also available
online. • 2009 Deane Root was appointed editor of Grove Music Online. By this time it contained
more than 50,000 articles. Grove is now considered to be the most comprehensive and scholarly source of information on Western Classical Art Music which exists. It is used daily by musicians and musicologists worldwide.
57
What might our queries be like in future? We will show some examples:
• A natural language phrase describes a musical feature.
• A regular expression or other pattern could not capture it.
58
Some Examples Q: 'horn call over shimmering strings'
59
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic"
60
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra'
61
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5
62
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5 Q: 'symphony featuring a battle between two timpanists'
63
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5 Q: 'symphony featuring a battle between two timpanists' A: Carl Nielsen, Symphony No. 4 "The Inextinguishable"
64
What are the Characteristics of these Queries? Not very long Not that specific ('over', 'featuring') Use musical terms ('horn', 'strings', 'stacatto', 'chords', 'full orchestra', 'symphony', 'timpani') Also use non-musical terms, interpreted in a musical way ('shimmering', 'battle')
65
What are the Characteristics of these Queries? Not very long Not that specific ('over', 'featuring') Use musical terms ('horn', 'strings', 'stacatto', 'chords', 'full orchestra', 'symphony', 'timpani') Also use non-musical terms, interpreted in a musical way ('shimmering', 'battle') Nevertheless, experts can answer them!