The C@merata task at MediaEval 2016: Natural Language Queries Derived from Exam Papers, Articles and Other Sources against Classical Music Scores in MusicXML Richard Sutcliffe, University of Essex Cynthia Liem, Delft University of Technology plus... Tom Collins, Lehigh University Eduard Hovy, Carnegie-Mellon University Richard Lewis, Goldsmiths, University of London Chris Fox, University of Essex Deane Root, University of Pittsburgh
74
Embed
MediaEval 2016 - the C@merata task: Natural Language Queries Derived from Exam Papers, Articles and Other Sources against Classical Music Scores in MusicXML
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
The C@merata task at MediaEval 2016: Natural Language Queries Derived
from Exam Papers, Articles and Other Sources against Classical Music Scores in MusicXML
Richard Sutcliffe, University of Essex
Cynthia Liem, Delft University of Technology
plus... Tom Collins, Lehigh University
Eduard Hovy, Carnegie-Mellon University Richard Lewis, Goldsmiths, University of London
Chris Fox, University of Essex Deane Root, University of Pittsburgh
2
Outline Applications Task Design Passages and Divisions Query Types and Music Scores Evaluation Metrics 2016 Campaign Results Conclusions
3
C@merata Task There is a series of 200 questions each year: • A short noun phrase in English referring to musical features in a score, • A short classical music score in MusicXML. Required Answer: • The location(s) in the score of the requested musical feature. We call each such location a passage.
4
Applications Linking musical references to scores: [ah25] repeated crotchet chords [dc364lh] tonic triad of E major [ah26] giant unison G from the entire orchestra Allowing experts to search scores interactively using phrases: triplet quavers against a chord in the bass Helping students to learn music theory: perfect cadence brings up an example in a score ah: Hopkins, A. (1982). The Nine Symphonies of Beethoven. London: Pan Books. dc: Cooke, D. (1995). Bruckner, (Joseph) Anton. In S. Sadie (ed), New Grove Dictionary of Music and Musicians, Volume 3, Section 7. Music (p362-366). London, UK: Macmillan.
5
Task Design There are 200 queries. We saw earlier that we have: Provided Question: • A short noun phrase in English referring to musical features in a score, • A short classical music score in MusicXML. Required Answer: • The location(s) in the score of the requested musical feature. We call each such location a passage.
6
Passages and Divisions We specify a passage as follows:
• a time signature, • a divisions value • a start bar and beat • an end bar and beat Example:
[4/4,1,1:1-2:4] time signature is 4/4. divisions value is 1 so we count in crotchets (quarter notes) Passage starts in bar 1 before the first crotchet (i.e. 1:1) Passage ends in bar two after the fourth crotchet (i.e. 2:4) Thus passage consists of the two complete bars (measures) numbered one and two.
7
What is a Divisions Value? The number of beats into which we divide a crotchet. A suitable value depends on what we wish to demarcate. Working in whole crotchets: Divisions = 1 Working in quavers: Divisions = 2 Working in quaver triplets: Divisions = 3 Working in semiquavers or quavers or triplet quavers: Divisions = 12 !! With Divisions = 12: one crotchet = 12 beats one quaver = 6 beats one quaver triplet = 4 beats one semiquaver = 3 beats Note: MusicXML uses the divisions concept and we got the idea from there.
8
Q: C D E F D E C in semiquavers repeated after a semiquaver A: [ 4/4, 4, 1:2-1:16 ] Q: semiquaver F# in the treble against a crotchet C in the bass A: [ 4/4, 4, 4:13-4:13 ]
9
Query Types In 2014 there were twelve types of query:
perf_spec 10 D sharp trill stave_spec 20 D4 in the right hand word_spec 5 word "Se" on an A flat
followed_by 30 crotchet followed by semibreve melodic_interval 19 melodic octave
harmonic_interval 11 harmonic major sixth cadence_spec 5 perfect cadence
triad_spec 5 tonic triad texture_spec 5 polyphony
All 200
10
In 2015 there were eight types of query, each modified in various ways:
Type No Example
1_melod 40 D4 minim eighth note in measure 9
1_melod qualified by
perf, instr, clef, time, key
40
trill on a quaver A G# in the Cello part in measures 29-39
sixteenth note C# in the left hand half note E3 in 2/2
sixteenth note G in G minor in measures 1-5
n_melod 20
F# E G F# A Do Mi Do Sol Do Mi Sol Do in bars 1-20
twenty semiquavers five note melody in bars 1-10
n_melod qualified by
perf, instr, clef, time, key
20
two staccato quarter notes in the Violin 1 crotchet, crotchet rest, crotchet rest, crotchet, crotchet rest, crotchet,
crotchet, crotchet, crotchet, crotchet in the Timpani melodic octave leap in the bass clef in measures 70-80
G4 B4 E5 in 3/4 rising G minor arpeggio
11
1_harm poss qualified by
perf, instr, clef, time, key
20
eighth note chord Bb, C, E chord of D minor in measures 109-110
harmonic minor sixth in the Violas dotted minim chord in the left hand
texture 6
monophonic passage homophony in measures 1-14 polyphony in measures 10-14 Alberti bass in measures 0-4
follow possibly qualified on
either or both sides by perf,
instr, clef, time, key
40
quavers F4 E4 in the oboe followed by quavers E2 G#2 in the bass clef quarter note minor third followed by eighth note unison
C followed by mordent Bb chord C4 G4 C5 E5 then a quaver
three eighth notes in the Violin I followed by twelve sixteenth notes in the Violin II in measures 87-92
synch possibly qualified in
either or both parts by perf,
instr, clef, time, key
14
four eighth notes against a half note crotchet D3 on the word “je” against a minim D2
four staccato quavers in the Violoncello against a minim chord Ab3 C4 F4 in the Harpsichord
All 200
12
In 2016 there are five types of query: 1_melod, n_melod, 1_harm, n_harm and texture 1_melod to n_harm can be qualified by perf, instr, clef, time, key All can be linked by follow or synch
Type No Example
1_melod 4 A#1 in bars 44-59 quarter-note rest in measures 1-5
1_melod qualified by perf, instr, clef, time,
key 36
dotted quarter note D6 in the first violin solo C5 in the oboe in measures 32 onwards
flute dotted half note only against strings half note on the tonic in the bass clef
A4 sung to the word 'bow'
n_melod 15
two-note dotted rhythm in measures 1-24 eight note rising passage in quarter notes
dotted minims C B A in the Bass clef in bars 70-90 melodic interval of a minor 7th in the voice
rising arpeggio in the left hand in measures 1-10 five-note melody in the cello in measures 20-28
whole note rest, quarter note in the Violin 4 in measures 1-103
1_harm 17
7th triad in measures 1-3 Ia chord in bars 1-10
chord of C whole-note unison E2 E3 E4
chord III in bars 44-59
1_harm possibly qualified by perf,
instr, clef, time, key 23
chord of F#3, D4 and A4 in the lower three parts harmonic fifth in the oboe
harmonic octave in the bass clef harmonic perfect fourth between the Soprano and Alto in bars 1-9
cello and viola playing dotted minims an octave apart in bars 40-70
14
n_harm 25
interrupted cadence A5 pedal in bars 116-138
authentic cadence in measures 14-18 plagal cadence in bars 134-138
three consecutive thirds in bars 1-43
n_harm possibly qualified by perf,
instr, clef, time, key 15
consecutive sixths between the Altos and Basses in measures 73-80 flute, oboe and bassoon in unison in measures 1-56
consecutive descending sixths in the left hand alternating fourths and fifths in the Oboe in bars 1-100
Soprano and Alto moving one step down together in measures 1-12
texture 20
all three violn parts in unison in measures 1-59 polyphony in measures 5-12
homophonic texture in measures 125-138 imitative texture in bars 1-18
counterpoint in bars 1-14 All 200
15
Type No Example
follow possibly qualified on either or both sides by perf,
instr, clef, time, key
20
C D E F D E C in semiquavers repeated after a semiquaver eighth-note twelfth followed by whole-note minor tenth between
Cello and Viola D C# in the right hand, then F A G Bb in semiquavers in the left
hand B flat in the cbass followed a quarter note later by B natural in the
cbass 5 B4s followed by a C5
synch possibly qualified in either or both parts by perf,
instr, clef, time, key
13
quarter note E5 against a quarter note C#3 C#3 minim and E4 semibreve simultaneously
D3 in the bass at the same time as C5 in soprano 1 three-note chord in the harpsichord right hand against a two-note
chord in the harpsichord left hand in measures 45-52 A#3 in the piano and F#5 in the voice simultaneously
16
Q: three-note chord in the harpsichord right hand against a two-note chord in the harpsichord left hand in measures 45-52 A: [ 3/4, 1, 45:3-45:3 ], [ 3/4, 1, 51:1-51:3 ]
17
Q: imitation in the piano left hand and voice in measures 1-6 A: [ 4/4, 1, 1:2-3:4 ]
handel_messiah_and_the_glory 18 fl, 2 ob, cl, bs, hn, tbn, tuba, SATB, hpd, 2 vn, va, vc, db Amer.
21
Evaluation of a Returned Passage Passage is beat correct if it starts at exactly the correct beat (as specified by the divisions value) in the correct start bar and also ends at the correct beat in the end bar. - Useful for applications of results which are themselves automatic. Passage is measure correct if it starts in the bar where the requested feature starts and ends in the bar where the requested feature ends. - Sufficient in many cases for humans to use.
22
Beat Precision (BP) is the number of beat correct passages returned by a system divided by the number of passages (correct or incorrect) returned. Beat Recall (BR) is the number of beat correct passages returned by a system divided by the total number of answer passages known to exist. Measure Precision (MP) is the number of measure correct passages returned by a system divided by the number of passages (correct or incorrect) returned. Measure Recall (MR) is the number of measure correct passages returned by a system divided by the total number of answer passages known to exist.
23
Participants
Runtag Leader Affiliation Country DMUN Andreas Katsiavalos De Montfort University England
KIAM Marina Mytrova Keldysh Institute of Applied Mathematics Russia
OMDN Donncha Ó Maidín University of Limerick Ireland
UMFC Paweł Cyrta Fryderyk Chopin University of Music Poland
Overall Conclusions We refined our ideas Some questions were chosen from real sources such as exam papers Questions got harder and hence more realistic This was difficult for participants!
31
We finish with examples of three interesting and complex question types: against follow texture
32
Q: dotted crotchet against quavers in the bass in bars 1-43 A: [ 3/4, 2, 9:3-9:5 ], [ 3/4, 2, 10:3-10:5 ], [ 3/4, 2, 11:3-11:5 ], [ 3/4, 2, 12:3-12:5 ]
33
Q: dotted crotchet followed by D5 A: [ 3/4, 4, 9:5-9:12 ], [ 3/4, 4, 11:5-11:12 ]
Example Questions The following slides show some example scores and questions against them.
37
Work: J.S. Bach, Das Wohltemperierte Klavier, Book 1, Prelude No. 2 in C minor BWV 847 Extract:
38
Q: passage in common time A: [4/4,1,1:1-4:4] Q: interval of a melodic sixth A: [4/4,4,1:1-1:2], [4/4,4,1:8-1:9] [4/4,4,1:9-1:10], [4/4,4,2:1-2:2], [4/4,4,2:8-2:9], [4/4,4,2:9-2:10], [4/4,4,3:1-3:2], [4/4,4,3:8-3:9], [4/4,4,3:9-3:10] Q: second A: None, because a second is harmonic by default Q: C followed by Eb A: [4/4,4,1:5-1:6], [4/4,4,1:13-1:14], [4/4,4,4:1-4:2], [4/4,4,4:9-4:10] Q: C followed by Eb in the bass clef A: [4/4,4,4:1-4:2], [4/4,4,4:9-4:10] Q: semiquaver E natural / sixteenth note E natural A: [4/4,4,2:3-2:4]
39
Work: J.S. Bach, Suite No. 3 in C Major for Cello, BWV 1009, Sarabande Extract:
Q: harmonic interval of a minor third A: [3/4,2,208:1-208:1] Q: minor third A: [3/4,2,208:1-208:1] (thirds are harmonic by default) Q: dotted quaver / dotted eighth note A: [3/4,4,206:5-206:7], [3/4,4,207:5-207:7], [3/4,4,208:5-208:7] Q: harmonic perfect fifth A: [3/4,1,206:1-206:1], [3/4,1,207:1-207:1], [3/4,4,208:5-208:7] (in last passage, C and G should be dotted but were not in the original) Q: simultaneous harmonic perfect fifth and harmonic eleventh A: [3/4,4,208:5-208:7] (same point, C and G are assumed dotted in this example)
40
Work: J. Dowland, King of Denmark's Galliard, P 40 Extract:
Q: perfect cadence A: [3/4,1,3:3-4:3] (We are assuming the cadence continues until the end of the bar) Q: four consecutive quavers / four consecutive quarter notes A: [3/4,2,2:3-2:6] Q: dotted minim in the bass / dotted half note in the bass A: [3/4,1,2:1-2:3] Q: harmonic fourth A: [3/4,1,1:1-1:1], [3/4,1,1:2-1:3], [3/4,1,4:1-4:3] (Note that there are two instances in bar (measure) 1 because the chord is played twice)
41
Work: J. Dowland, Pauana Dulandi, P 86 Extract:
Q: consecutive 5ths in the bass A: [4/4,1,2:1-2:4] Q: semibreve Bb in the treble clef / whole note Bb in the treble clef A: [4/4,1,1:1-1:4], [4/4,1,4:1-4:4] Q: octave followed two bars later by another octave A: [4/4,1,1:1-3:4] (assumed harmonic) Q: Vc triad A: [4/4,1,2:3-2:3]
42
Work: G.F. Handel, Messiah, HG xlv, "And the Glory of the Lord" Extract:
Q: crotchet rest / quarter note rest A: [3/4,1,1:1-1:1] Q: dotted crotchet followed by three quavers / dotted quarter note followed by three eighth notes A: [3/4,1,5:1-5:3], [3/4,1,6:1-6:3] Q: four hemidemisemiquavers / four sixty-fourth notes A: [3/4,4,3:9-3:9] Q: three quavers in a row / three eighth notes in a row A: [3/4,2,5:4-5:6], [3/4,2,6:4-6:6]
Q: D sharp crotchet / D sharp quarter note A: [4/4,1,22:1-22:1], [4/4,1,25:1-25:1] Q: D natural quaver / D natural eighth note A: [4/4,2,23:3-23:3], [4/4,2,24:4-24:4] Q: open VII triad in the first inversion A: [4/4,1,23:3-23:3] Q: melodic fourth in the bass clef A: [4/4,1,24:1-24:2]
44
Work: D. Scarlatti, Keyboard Sonata in D minor, K 1 Extract:
45
Q: eight staccato notes in succession A: [4/4,1,17:1-17:4] Q: augmented melodic fourth A: [4/4,2,19:3-19:4], [4/4,2,20:4-20:5] Q: third A: [4/4,4,17:13-17:13], [4/4,2,19:2-19:2], [4/4,2,19:4-19:4], [4/4,4,19:9-19:9], [4/4,4,20:9-20:9] (these are assumed to be harmonic and can thus be across parts) Q: a quaver, then a major third / an eighth note, then major third A: [4/4,2,19:1-19:2], [4/4,2,19:3-19:4] (third assumed to be harmonic and to follow the quaver immediately) Q: change from bass to treble clef A: [4/4,1,17:2p] (recall that this means the clef change is immediately after 17:2)
46
Work: D. Scarlatti, Keyboard Sonata in D major, K 430 Extract:
Q: three fourths A: [3/8,2,56:1-56:3] (assumed consecutive) Q: treble clef F natural A: [3/8,2,62:2-62:2] Q: melodic octave A: [3/8,4,57:1-57:2], [3/8,4,57:2-57:6] (melodic must be stated, otherwise it is harmonic) Q: harmonic 5th followed by harmonic 4th A: [3/8,2,58:1-58:2], [3/8,2,62:1-62:2]
47
Architecture of Baseline System We have built a very basic system to perform the task: 1. Analyse the input xml file and extract questions and score files 2. For each question: • Parse question using Stanford Parser • Determine question type (similar to normal QA) • Parse the score file using Music21 • Based on question type, search for answer passages in score 3. Write out answers to output xml file
48
Linguistic Observations Queries are noun phrases Head noun group is the main feature (e.g. F#) PP modifiers qualify this (e.g. in the bass clef) A lot of terminology is used (e.g. semiquaver, soprano part, forte bar) Search based on query classification will work at least for simple examples.
49
Summary We are trying to find ways to link natural language descriptions of music to musical scores. Starting with the C@merata task, we are working with very simple tasks to develop the technology. Then we will progress to more complex tasks which are of genuine interest to musicologists. In due course, we aim to tie passages in musicological texts such as in Grove Online to the corresponding music scores.
50
Music Extracts symphony opening with a horn call over shimering strings Bruckner 4th Symphony http://www.youtube.com/watch?v=J8t1TzN0RRY Start: 0:00 End: 0:44 symphony closing with six unison chords Sibelius 5th Symphony http://www.youtube.com/watch?v=nkzrSZKA4cM Start: 9:36 End: 9:56 Nielsen 4th Symphony, Finale (shows two timpanists well) http://www.youtube.com/watch?v=yXDe1hj4HBo Start: 32:10 End: 32:48
51
Extracts Not Used symphony opening with two octave descending arpeggio Beethoven 9th Symphony string quartet opening including double stopping on all four instruments Beethoven op 127 symphony featuring a battle between two timpanists Nielsen 4th Symphony, Finale (not so good) http://www.youtube.com/watch?v=sD9I-UiYfW8 Start: 6:30 End: 7:08
52
Grove’s Dictionary of Music and Musicians • 1879-1889 George Grove, civil engineer, music administrator, writer and then Director of
Royal College of Music, wrote A Dictionary of Music and Musicians in four volumes. • 1904-910 Fuller Maitland edited the second edition - Grove's Dictionary of Music and
Musicians - in five volumes. • 1927 Henry Colles edited the third edition - in five volumes. • 1940 Henry Colles edited the fourth edition - in seven volumes. • 1954 Eric Blom edited the fifth edition - in nine volumes.
53
• 1980 Stanlie Sadie edited the sixth edition - The New Grove Dictionary of Music and Musicians - in twenty volumes. Contained 22,500 articles and 16,500 biographies.
• 2001 Stanlie Sadie edited the seventh edition - in 29 volumes. This was also available online. • 2009 Deane Root was appointed editor of Grove Music Online. By this time it contained more
than 50,000 articles. Grove is now considered to be the most comprehensive and scholarly source of information on Western Classical Art Music which exists. It is used daily by musicians and musicologists worldwide.
54
What might our queries be like in future? We will show some examples: • A natural language phrase describes a musical feature.
• A regular expression or other pattern could not capture it.
55
Some Examples Q: 'horn call over shimmering strings'
56
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic"
57
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra'
58
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5
59
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5 Q: 'symphony featuring a battle between two timpanists'
60
Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5 Q: 'symphony featuring a battle between two timpanists' A: Carl Nielsen, Symphony No. 4 "The Inextinguishable"
61
What are the Characteristics of these Queries? Not very long Not that specific ('over', 'featuring') Use musical terms ('horn', 'strings', 'stacatto', 'chords', 'full orchestra', 'symphony', 'timpani') Also use non-musical terms, interpreted in a musical way ('shimmering', 'battle')
62
What are the Characteristics of these Queries? Not very long Not that specific ('over', 'featuring') Use musical terms ('horn', 'strings', 'stacatto', 'chords', 'full orchestra', 'symphony', 'timpani') Also use non-musical terms, interpreted in a musical way ('shimmering', 'battle') Nevertheless, experts can answer them!