Stravinsqi/De Montfort University at the MediaEval 2014 C@merata Task

Tom Collins Faculty of Technology

www.tomcollinsresearch.net

•  Stravinsqi stands for STaff Representation Analysed VIa Natural language String Query Input.

1. Get question string and division value, analyse it and split into compound questions if necessary.

2. Load staff names, time signatures, and point-‐set representations of the piece.

3. Produce time-‐interval answers to elemental question(s). 4. Cross-‐check timings of answers to compound questions. 5. Convert time-‐interval answers to required format.





















; Set paths.

(setq

question-‐path&name

(merge-‐pathnames

(make-‐pathname

:directory

'(:relative

"C@merata2014" "training_v1")

:name "training_v1" :type "xml")

*MCStylistic-‐MonthYear-‐data-‐path*))

(setq

notation-‐path

(merge-‐pathnames

(make-‐pathname

:directory

'(:relative

"C@merata2014" "training_v1"))

*MCStylistic-‐MonthYear-‐data-‐path*))

#| Let's begin by looking at Morley's 'April is in my

mistress' face'. |#

(setq notation-‐name "f1")

(setq

notation-‐path&name

(merge-‐pathnames

(make-‐pathname

:name notation-‐name :type "krn") notation-‐path))

(setq question-‐number "001")

#| *********************************

**** 1. ANALYSE QUESTION STRING ****

********************************* |#

; Load question string and division value.

(setq

question&division

(c@merata2014-‐question-‐file2question-‐string

question-‐number question-‐path&name

notation-‐name))

; -‐-‐> ("F# followed two crotchets later by a G" 1)

(setq division (second question&division))

; -‐-‐> 1

; Split up compound questions.

(setq questions

(followed-‐by-‐splitter

(first question&division)))

; -‐-‐> (("F#" 0) ("G" 2))

; Try US version of the question string.

(setq questions


"F# followed two quarter notes later by a G"))

; -‐-‐> (("F#" 0) ("G" 2))

; Try junk version of the question string.

(setq questions


"F# blah followed two crotchets later by a blah G"))

; -‐-‐> (("F# blah" 0) ("blah G" 2))

; Try reference to bars rather than a specific duration.

(setq questions


"F# followed two bars later by a G"))

; -‐-‐> (("F#" 0) ("G" 2 "bars"))

; Try 'consecutive' question.

(setq questions


"F# followed by two consecutive crotchets"))

; -‐-‐> (("F#" 0) ("crotchet" 0) ("crotchet" 0))

; Try linguistic variant of compound query.

(setq questions


(concatenate

'string "F# then two bars later a G "

"then consecutive fifths in the left hand")))

#|

-‐-‐> (("F#" 0) ("G" 2 "bars")

("fifths in the left hand" 0)

("fifths in the left hand" 0))

|#

; Back to original question.

(setq questions


(first question&division)))

; -‐-‐> (("F#" 0) ("G" 2))

#| ***************************************

**** 2. LOAD REPRESENTATIONS OF PIECE ****

*************************************** |#

#| Load staff names and names of the opening clefs on

each staff. |#

(setq

staff&clef-‐names

(staves-‐info2staff&clef-‐names notation-‐path&name))

#|

-‐-‐> (("Bass" "bass clef") ("Tenor" "tenor clef")

("Alto" "treble clef") ("Soprano" "treble clef"))

|#

#| Load time-‐signature information and any changes

across the piece. |#

(setq

ontimes-‐signatures

(append-‐ontimes-‐to-‐time-‐signatures

(kern-‐file2ontimes-‐signatures notation-‐path&name)))

; -‐-‐> ((1 4 4 0))

; If there was a change, it might look like:

; -‐-‐> ((1 4 4 0) (9 3 2 32))

; Load point-‐set representations.

(setq

artic-‐set

(kern-‐file2points-‐artic-‐dynam-‐lyrics

notation-‐path&name))

#|

-‐-‐> ((0 46 52 4 3 NIL NIL ("place."))

(0 58 59 2 2 NIL NIL ("place."))

(0 65 63 2 1 NIL NIL ("place."))

(0 70 66 2 0 NIL NIL ("place"))

(3 58 59 1 2 NIL NIL ("With-‐")) ...)

|#

#| **********************************************

**** 3. TIME-‐INTERVAL ANSWERS TO QUESTION(S) ****

********************************************** |#

(setq

ans-‐harmonic-‐interval

(mapcar

#'(lambda (x)

(harmonic-‐interval-‐of-‐a

(first x) point-‐set staff&clef-‐names))

questions))

; -‐-‐> (NIL NIL)

; Mel. intervals to be prefaced by "melodic".

(setq

ans-‐melodic-‐interval

(mapcar

#'(lambda (x)

(melodic-‐interval-‐of-‐a


questions))

; -‐-‐> (NIL NIL)

(setq

ans-‐dur-‐pitch

(mapcar

#'(lambda (x)

(duration&pitch-‐class-‐time-‐intervals


questions))

; -‐-‐> (NIL NIL)

; This is the function of interest on this occasion...

(setq

ans-‐pitch

(mapcar

#'(lambda (x)

(pitch-‐class-‐time-‐intervals


questions))

#|

-‐-‐> (((32 36) (57 115/2) (58 60))

((12 13) (12 13) (24 32) (26 27) (28 32) (46 48)

(48 50) (49 50) (50 52) (50 52) (54 57) (60 62) (60 61) (61 62) (62 63) (63 64)))

|#

#| ****************************************************

**** 4. Cross-‐check timings for compound questions ****

**************************************************** |#

; Put the answers next to one another in one list.

(setq

time-‐intervals

(list

ans-‐harmonic-‐interval ans-‐melodic-‐interval

ans-‐dur-‐pitch ans-‐pitch ans-‐dur ans-‐nadir-‐apex

ans-‐triad ans-‐triad-‐inversion ans-‐texture ans-‐cadence

ans-‐word&event ans-‐word ans-‐artic ans-‐dur-‐rest

ans-‐tied&event))

#|

-‐-‐> ((NIL NIL) (NIL NIL) (NIL NIL)

(((32 36) (57 115/2) (58 60))

((12 13) (12 13) (24 32) (26 27) (28 32) (46 48)

(48 50) (49 50) (50 52) (50 52) (54 57) (60 62)

(60 61) (61 62) (62 63) (63 64) (63 64)))

(NIL NIL) (NIL NIL) (NIL NIL) (NIL NIL) (NIL NIL)

(NIL NIL) (NIL NIL) (NIL NIL) (NIL NIL) (NIL NIL)

(NIL NIL))

|#

; Cross-‐check the timings of compound questions.

(setq

time-‐intervals

(if (> (length questions) 1)

(cross-‐check-‐compound-‐questions

questions ontimes-‐signatures time-‐intervals)

(if (index-‐item-‐1st-‐doesnt-‐occur (list nil) time-‐intervals)

(first

(nth (index-‐item-‐1st-‐doesnt-‐occur (list nil) time-‐intervals) time-‐intervals)))))

; -‐-‐> ((58 62))

(bar&beat-‐number-‐of-‐ontime 58 ontimes-‐signatures)

; -‐-‐> (15 3 58)

Spider Plot

0.25 0.5 0.75 1All0.25

0.5

0.75

1

Pitch

0.25

0.5

0.75

1

Duration

0.25

0.5

0.75

1

Pitch and Duration

0.25

0.5

0.75

1Articulation

0.25

0.5

0.75

1Voice Specific

0.250.5

0.751

Lyrics

0.250.5

0.751

Compound

0.25

0.5

0.75

1

Melodic Interval

0.25

0.5

0.75

1

Harmonic Interval

0.25

0.5

0.75

1Cadence

0.25

0.5

0.75

1 Triad

0.25

0.5

0.75

1 Texture

Measure RecallMeasure PrecisionBeat RecallBeat Precision

Triad Spider Plot

0.25 0.5 0.75 1All0.25

0.5

0.75

1

Pitch

0.25

0.5

0.75

1

Duration

0.25

0.5

0.75

1

Pitch and Duration

0.25

0.5

0.75

1Articulation

0.25

0.5

0.75

1Voice Specific

0.250.5

0.751

Lyrics

0.250.5

0.751

Compound

0.25

0.5

0.75

1

Melodic Interval

0.25

0.5

0.75

1

Harmonic Interval

0.25

0.5

0.75

1Cadence

0.25

0.5

0.75

1 Triad

0.25

0.5

0.75

1 Texture

Measure RecallMeasure PrecisionBeat RecallBeat Precision

Mean

(melody with accompn, 1)!

(contrapuntal, 1)!

(melody with accompn, .8)!

•  The Stravinsqi algorithm has effectively solved seven of the twelve C@merata task categories (pitch, duration, pitch and duration, articulation, voice specific, lyrics, and melodic interval)

•  As for the remaining five categories, precision for one category (compound queries) can be improved by fixing a selection-‐criteria bug

•  More data are required for the triad, cadence, and texture query categories

•  A big “thank you” to Richard Sutcliffe and team













Researchers at CCARH for creating and hosting kern scores

Collaborators at:

Johannes Kepler University: Gerhard Widmer, Andreas Arzt, Sebastian Böck, and Sebastian Flossmann

Center for Mind and Brain, UC Davis: Petr Janata, Fred Barrett, and Joy Geng

Aalborg University: David Meredith

University of Lyon: Barbara Tillmann

The Open University: Robin Laney, Alistair Willis, and Paul Garthwaite

Funding bodies: Austrian Science Fund (FWF) project number Z159 (Wittgenstein Grant), NSF grant #1025310, ESPRC

Code for Stravinsqi (as well as other projects):

http://www.tomcollinsresearch.net

!"

!"

Allegro non tanto

# $$$$$ %%!

& % !

& ! & & & '% !

& ! & & & %!

& ! & & & '%!

& ! & & ! &' (

) $$$$$ * &+ & & & & & & * &+ & & &' & & & * &'+ & &

6

# $$$$$ % !

& !

& & &' '%!

& ! & &' &&' '%& !

&' % &' '%!

& ! & & ! &' ( % !

& !

& & &'

) $$$$$ ,%' ! %%'''% !&& %%''

% !&& %%'' %%!

!

* &' ( & & '% !&& %%''

11

# $$$$$ '%!

& ! & &' &&' %$*!%

!

&- &. && ''$ &&% !

& & &% & & / & & & & / & & & &&&(

) $$$$$ '% !&& %%''&&& &' & &&

& ! &+ %& / & % & %& / & % & &(

!

P1,2

!

P2,2

!

P3,3

!

P3,2

!

P2,1

!

P3,4

!

P2,3

!

P3,1

!

P1,1

Beginning of op.56 no.1 by Chopin

!"

!"

Allegro non tanto

# $$$$$ % %& %&' $$$$$

% %&

6

# $$$$$

' $$$$$ %&

11

# $$$$$

' $$$$$

!

P22

!

P33

!

P32

!

P21

!

P34

!

P23

!

P31

!

P11

!

P12

Beginning of op.56 no.1 by Chopin

!"

!"

Allegro non tanto

# $$$$$% % % % % % % & % ''

'%' % ! %%%% % % %' % % % ( %' %' % %' %'

3

3

3

# $$$$$ %%%' %%' %%% %%%% %!

( )%%$'&%' % !* &# %%' %%%' %%%'' % & +'%' %' %%%% !* & %%'' %%%''

6

# $$$$$ %' % %' % ! %' %' %' % ''+ %

%%% %%'+ '''%' & %' %%%% %%%!

% %' %' % %' %' %' % %' % ! %'3

# $$$$$%' % %%' %% ) %'

% ( %' ( ( # %%%+' %%'' %%%'' %' % %%' %% )

11

# $$$$$ %' %' % ''+ %

% (%% %%'+ '''%' & %' %%%% %%% $% !

% %%* , ,3

% % ! %- %%%$- %%%$-- %' %' % % %$) $$$$$ %'

% ( %' ( ( %%%$$$ %%%$$$ (%%$$ %%-- %%'$ %%'

%%%%

!"#"#"

#"$" %"

&" &"

&"

Generated passage

Code for Stravinsqi (as well as other projects):

http://www.tomcollinsresearch.net

Stravinsqi/De Montfort University at the MediaEval 2014 C@merata Task

Software