Top Banner
Functional Modelling of Musical Harmony And Its Applications Jos´ e Pedro Magalh˜ aes Department of Computer Science, University of Oxford December 9, 2014 72nd IFIP 2.1 Meeting Vermont, United States of America Jos´ e Pedro Magalh˜ aes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 1 / 46
80

Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Jul 30, 2018

Download

Documents

trandieu
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Functional Modelling of Musical HarmonyAnd Its Applications

Jose Pedro Magalhaes

Department of Computer Science, University of Oxford

December 9, 201472nd IFIP 2.1 Meeting

Vermont, United States of America

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 1 / 46

Page 2: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Introduction

I Quick introduction to musical harmony

I Modelling musical harmony using Haskell

I Applications of a model of harmony:I Musical analysisI Musical similarityI Generating chords and melodiesI Correcting errors in chord extractionI Chordify—a web-based music player with chord recognitionI Merging chord edits—another application?

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 2 / 46

Page 3: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Table of Contents

Harmony

A functional model of harmony

Harmony analysis

Harmonic similarity

Music generation

Chord recognition: Chordify

Crowd-sourcing chord edits

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 3 / 46

Page 4: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

What is harmony?

IVIVI V/V

C F D G7 7 C

{Ton TonSDom Dom

I Harmony arises when at least two notes sound at the same time

I It’s the “vertical” aspect of music (with melody being the“horizontal” aspect)

I Harmony induces tension and release patterns, that can be describedby music theory and music cognition

I The surrounding context also has a large influence

I It is generally highly structured, and obeys musical composition rules

Demo: how harmony affects melody

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 4 / 46

Page 5: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

What is harmony?

IVIVI V/V

C F D G7 7 C

{Ton TonSDom Dom

I Harmony arises when at least two notes sound at the same time

I It’s the “vertical” aspect of music (with melody being the“horizontal” aspect)

I Harmony induces tension and release patterns, that can be describedby music theory and music cognition

I The surrounding context also has a large influence

I It is generally highly structured, and obeys musical composition rules

Demo: how harmony affects melody

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 4 / 46

Page 6: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Simplified harmony theory I

I A chord is a group of tones separated by intervals of roughly thesame size.

I All music is made out of chords (whether explicitly or not).

I There are 12 different notes. Instead of naming them, we numberthem relative to the first and most important one, the tonic. So weget I, II[, II . . . VI], VII.

I A chord is built on a root note. So I also stands for the chord builton the first degree, V for the chord built on the fifth degree, etc.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 5 / 46

Page 7: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Simplified harmony theory II

Models for musical harmony explain the harmonic progression in music:

I Everything works around the tonic (I).

I The dominant (V) leads to the tonic.

I The subdominant (IV) tends to lead to the dominant.

I Therefore, the I IV V I progression is very common.

I There are also secondary dominants, which lead to a relative tonic.For instance, II7 is the secondary dominant of V, and I7 is thesecondary dominant of IV.

I So you can start with I, add one note to get I7, fall into IV, changetwo notes to get to II7, fall into V, and then finally back to I.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 6 / 46

Page 8: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

An example harmonic analysis

IVIVI V/V

C F D G7 7 C

{Ton TonSDom Dom

Piece

PT

T

I

C

PD

D

D

V/V

V7

G:7

II7

D:7

S

IV

F

PT

T

I

C

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 7 / 46

Page 9: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Why are harmony models useful?

Having a model for musical harmony allows us to automaticallydetermine the functional meaning of chords in the tonal context.

The model determines which chords “fit” on a particular moment in asong. This is very useful, as we will see.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 8 / 46

Page 10: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Table of Contents

Harmony

A functional model of harmony

Harmony analysis

Harmonic similarity

Music generation

Chord recognition: Chordify

Crowd-sourcing chord edits

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 9 / 46

Page 11: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

A functional model of harmony

PieceM → [PhraseM] (M ∈ {Maj,Min})

PhraseM → TonM DomM TonM

| DomM TonM

TonMaj → IMaj

TonMin → ImMin

DomM → V7M

| VM

| VII0M| SubM DomM

| II7M V7M

SubMaj → IImMaj

| IVMaj

| IIImMaj IVMaj

SubMin → IVmMin

Simple, but enough for now, and easy to extend.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 10 / 46

Page 12: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

A functional model of harmony

PieceM → [PhraseM] (M ∈ {Maj,Min})

PhraseM → TonM DomM TonM

| DomM TonM

TonMaj → IMaj

TonMin → ImMin

DomM → V7M

| VM

| VII0M| SubM DomM

| II7M V7M

SubMaj → IImMaj

| IVMaj

| IIImMaj IVMaj

SubMin → IVmMin

Simple, but enough for now, and easy to extend.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 10 / 46

Page 13: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

A functional model of harmony

PieceM → [PhraseM] (M ∈ {Maj,Min})

PhraseM → TonM DomM TonM

| DomM TonM

TonMaj → IMaj

TonMin → ImMin

DomM → V7M

| VM

| VII0M| SubM DomM

| II7M V7M

SubMaj → IImMaj

| IVMaj

| IIImMaj IVMaj

SubMin → IVmMin

Simple, but enough for now, and easy to extend.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 10 / 46

Page 14: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

A functional model of harmony

PieceM → [PhraseM] (M ∈ {Maj,Min})

PhraseM → TonM DomM TonM

| DomM TonM

TonMaj → IMaj

TonMin → ImMin

DomM → V7M

| VM

| VII0M| SubM DomM

| II7M V7M

SubMaj → IImMaj

| IVMaj

| IIImMaj IVMaj

SubMin → IVmMin

Simple, but enough for now, and easy to extend.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 10 / 46

Page 15: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

A functional model of harmony

PieceM → [PhraseM] (M ∈ {Maj,Min})

PhraseM → TonM DomM TonM

| DomM TonM

TonMaj → IMaj

TonMin → ImMin

DomM → V7M

| VM

| VII0M| SubM DomM

| II7M V7M

SubMaj → IImMaj

| IVMaj

| IIImMaj IVMaj

SubMin → IVmMin

Simple, but enough for now, and easy to extend.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 10 / 46

Page 16: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

A functional model of harmony

PieceM → [PhraseM] (M ∈ {Maj,Min})

PhraseM → TonM DomM TonM

| DomM TonM

TonMaj → IMaj

TonMin → ImMin

DomM → V7M

| VM

| VII0M| SubM DomM

| II7M V7M

SubMaj → IImMaj

| IVMaj

| IIImMaj IVMaj

SubMin → IVmMin

Simple, but enough for now, and easy to extend.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 10 / 46

Page 17: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—IA naive datatype encoding musical harmony:

data Piece = Piece [ Phrase ]

data Phrase wherePhraseIVI :: Ton→ Dom→ Ton→ PhrasePhraseVI :: Dom→ Ton→ Phrase

data Ton whereTonMaj :: Degree→ TonTonMin :: Degree→ Ton

data Dom whereDomV7 :: Degree→ DomDomV :: Degree→ DomDomVII0 :: Degree→ DomDomIV−V :: SDom→ Dom→ DomDomII−V :: Degree→ Degree→ Dom

data Degree = I | II | III . . .

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 11 / 46

Page 18: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—IA naive datatype encoding musical harmony:

data Piece = Piece [ Phrase ]

data Phrase wherePhraseIVI :: Ton→ Dom→ Ton→ PhrasePhraseVI :: Dom→ Ton→ Phrase

data Ton whereTonMaj :: Degree→ TonTonMin :: Degree→ Ton

data Dom whereDomV7 :: Degree→ DomDomV :: Degree→ DomDomVII0 :: Degree→ DomDomIV−V :: SDom→ Dom→ DomDomII−V :: Degree→ Degree→ Dom

data Degree = I | II | III . . .

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 11 / 46

Page 19: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—IA naive datatype encoding musical harmony:

data Piece = Piece [ Phrase ]

data Phrase wherePhraseIVI :: Ton→ Dom→ Ton→ PhrasePhraseVI :: Dom→ Ton→ Phrase

data Ton whereTonMaj :: Degree→ TonTonMin :: Degree→ Ton

data Dom whereDomV7 :: Degree→ DomDomV :: Degree→ DomDomVII0 :: Degree→ DomDomIV−V :: SDom→ Dom→ DomDomII−V :: Degree→ Degree→ Dom

data Degree = I | II | III . . .

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 11 / 46

Page 20: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—IA naive datatype encoding musical harmony:

data Piece = Piece [ Phrase ]

data Phrase wherePhraseIVI :: Ton→ Dom→ Ton→ PhrasePhraseVI :: Dom→ Ton→ Phrase

data Ton whereTonMaj :: Degree→ TonTonMin :: Degree→ Ton

data Dom whereDomV7 :: Degree→ DomDomV :: Degree→ DomDomVII0 :: Degree→ DomDomIV−V :: SDom→ Dom→ DomDomII−V :: Degree→ Degree→ Dom

data Degree = I | II | III . . .

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 11 / 46

Page 21: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—II

A GADT encoding musical harmony:

data Mode = MajMode | MinMode

data Piece (µ :: Mode) wherePiece :: [ Phrase µ ]→ Piece µ

Advanced functional programming begins here; we’re using datatypepromotion to constrain the shape of our indices.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 12 / 46

Page 22: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—III

The rest of the model mimicks the context-free grammar shown before:

data Phrase (µ :: Mode) wherePhraseIVI :: Ton µ→ Dom µ→ Ton µ→ Phrase µPhraseVI :: Dom µ→ Ton µ→ Phrase µ

data Ton (µ :: Mode) whereTonMaj :: SD I Maj→ Ton MajMode

TonMin :: SD I Min→ Ton MinMode

data Dom (µ :: Mode) whereDomV7 :: SD V Dom7 → Dom µDomV :: SD V Maj → Dom µDomVII0 :: SD VII Dim → Dom µDomIV−V :: SDom µ→ Dom µ→ Dom µDomII−V :: SD II Dom7 → SD V Dom7 → Dom µ

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 13 / 46

Page 23: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—III

The rest of the model mimicks the context-free grammar shown before:

data Phrase (µ :: Mode) wherePhraseIVI :: Ton µ→ Dom µ→ Ton µ→ Phrase µPhraseVI :: Dom µ→ Ton µ→ Phrase µ

data Ton (µ :: Mode) whereTonMaj :: SD I Maj→ Ton MajMode

TonMin :: SD I Min→ Ton MinMode

data Dom (µ :: Mode) whereDomV7 :: SD V Dom7 → Dom µDomV :: SD V Maj → Dom µDomVII0 :: SD VII Dim → Dom µDomIV−V :: SDom µ→ Dom µ→ Dom µDomII−V :: SD II Dom7 → SD V Dom7 → Dom µ

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 13 / 46

Page 24: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—III

The rest of the model mimicks the context-free grammar shown before:

data Phrase (µ :: Mode) wherePhraseIVI :: Ton µ→ Dom µ→ Ton µ→ Phrase µPhraseVI :: Dom µ→ Ton µ→ Phrase µ

data Ton (µ :: Mode) whereTonMaj :: SD I Maj→ Ton MajMode

TonMin :: SD I Min→ Ton MinMode

data Dom (µ :: Mode) whereDomV7 :: SD V Dom7 → Dom µDomV :: SD V Maj → Dom µDomVII0 :: SD VII Dim → Dom µDomIV−V :: SDom µ→ Dom µ→ Dom µDomII−V :: SD II Dom7 → SD V Dom7 → Dom µ

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 13 / 46

Page 25: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—IV

Scale degrees are the leaves of our hierarchical structure:

data DiatonicDegree = I | II | III | IV | V | VI | VIIdata Quality = Maj | Min | Dom7 | Dim

data SD (δ :: DiatonicDegree) (γ :: Quality) whereSurfaceChord :: ChordDegree→ SD δ γ

Now everything is properly indexed, and our GADT is effectivelyconstrained to allow only “harmonically valid” sequences!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 14 / 46

Page 26: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—IV

Scale degrees are the leaves of our hierarchical structure:

data DiatonicDegree = I | II | III | IV | V | VI | VIIdata Quality = Maj | Min | Dom7 | Dim

data SD (δ :: DiatonicDegree) (γ :: Quality) whereSurfaceChord :: ChordDegree→ SD δ γ

Now everything is properly indexed, and our GADT is effectivelyconstrained to allow only “harmonically valid” sequences!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 14 / 46

Page 27: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—V

We don’t export SurfaceChord; these are built with a smart constructor:

deg :: (ToDegree δ,ToQuality γ)⇒ Proxy δ → Proxy γ → SD δ γdeg d q = SurfaceChord (toDegree d) (toQuality q)

We need type-to-value conversions for degrees:

class ToDegree δ where toDegree :: Proxy δ → Degree

instance ToDegree I where toDegree = Iinstance ToDegree II where toDegree = II. . . -- instances for all degrees

(Type-to-value conversion actually handled automatically if we use thesingletons package.)

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 15 / 46

Page 28: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Now in Haskell—VI

Now terms like TonMaj (deg (Proxy :: Proxy II) (Proxy :: Proxy Maj)) donot typecheck, as we wanted:

ghci> Ton_Maj (deg (Proxy :: Proxy II) (Proxy :: Proxy Maj))

Couldn’t match expected type ‘I’

with actual type ‘II’

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 16 / 46

Page 29: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Increasing complexity

The model we presented so far is very simple and cannot account for:

Repeated chords GMaj GMaj CMaj

Diatonic secondary dominants Emin Amin Dmin G7 CMaj

Chromatic secondary dominants A7 D7 G7 CMaj

Minor key dominant borrowing Gmin CMaj

Tritone substitutions A[7 G7 CMaj

. . .

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 17 / 46

Page 30: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Handling secondary dominants

More fun: type families for computing perfect fifths:

data BDegSD (δ :: DiatonicDegree) whereCons :: BDegSD (PerfV δ)→ Degree δ → BDegSD δBase :: Degree δ → BDegSD δ

type family PerfV (δ :: DiatonicDegree) :: DiatonicDegree

type instance PerfV I = Vtype instance PerfV V = II. . .

I could go on for a while, but let’s move on to actual applications of thismodel.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 18 / 46

Page 31: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Handling secondary dominants

More fun: type families for computing perfect fifths:

data BDegSD (δ :: DiatonicDegree) whereCons :: BDegSD (PerfV δ)→ Degree δ → BDegSD δBase :: Degree δ → BDegSD δ

type family PerfV (δ :: DiatonicDegree) :: DiatonicDegree

type instance PerfV I = Vtype instance PerfV V = II. . .

I could go on for a while, but let’s move on to actual applications of thismodel.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 18 / 46

Page 32: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Table of Contents

Harmony

A functional model of harmony

Harmony analysis

Harmonic similarity

Music generation

Chord recognition: Chordify

Crowd-sourcing chord edits

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 19 / 46

Page 33: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Application: displaying harmony structurally

Parsing the sequence Gmin C7 Gmin C7 FMaj D7 G7 CMaj:

PhraseVI

TonMaj

I

C:maj

DomIV−V

DomIV−V

DomII−V

V7

G:7

V/V

II7

D:7

Sub

IV

F:maj

V/IV

I7

C:7

V/I

Vmin

G:min

Sub

IV

ins

V/IV

I7

C:7

V/I

Vmin

G:min

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 20 / 46

Page 34: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Parsing chord sequences I

We now consider the problem of parsing (textual) chord sequences intoour datatype representing musical harmony:

class Parse α whereparse :: Parser α

We use Doaitse’s amazing error-correcting parser combinators, and weget harmony-compliant chord sequences for free!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 21 / 46

Page 35: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Parsing chord sequences I

We now consider the problem of parsing (textual) chord sequences intoour datatype representing musical harmony:

class Parse α whereparse :: Parser α

We use Doaitse’s amazing error-correcting parser combinators, and weget harmony-compliant chord sequences for free!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 21 / 46

Page 36: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Parsing chord sequences II

Most instances are trivial:

instance Parse (Piece µ) whereparse = Piece <$> parse

instance Parse (Phrase µ) whereparse = PhraseIVI <$> parse

<|> PhraseVI <$> parse

instance Parse (Ton MajMode) whereparse = TonMaj <$> parse

instance Parse (Ton MinMode) whereparse = TonMin <$> parse

. . .

So trivial that we do not write them; we use a generic parser.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 22 / 46

Page 37: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Parsing chord sequences II

Most instances are trivial:

instance Parse (Piece µ) whereparse = Piece <$> parse

instance Parse (Phrase µ) whereparse = PhraseIVI <$> parse

<|> PhraseVI <$> parse

instance Parse (Ton MajMode) whereparse = TonMaj <$> parse

instance Parse (Ton MinMode) whereparse = TonMin <$> parse

. . .

So trivial that we do not write them; we use a generic parser.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 22 / 46

Page 38: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Parsing chord sequences III

Two challenges to triviality:

1. How to do generic programming with indexed datatypes

I Current libraries can’t do it (Uniplate, SYB, GHC.Generics, etc.)I So we came up with oneI . . . which kind of works.

2. How to instantiate (generic) functions to indexed datatypesI Consumers (like show) are easyI Producers (like read, or our parse) are trickier!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 23 / 46

Page 39: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Parsing chord sequences III

Two challenges to triviality:

1. How to do generic programming with indexed datatypesI Current libraries can’t do it (Uniplate, SYB, GHC.Generics, etc.)I So we came up with one

I . . . which kind of works.

2. How to instantiate (generic) functions to indexed datatypesI Consumers (like show) are easyI Producers (like read, or our parse) are trickier!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 23 / 46

Page 40: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Parsing chord sequences III

Two challenges to triviality:

1. How to do generic programming with indexed datatypesI Current libraries can’t do it (Uniplate, SYB, GHC.Generics, etc.)I So we came up with oneI . . . which kind of works.

2. How to instantiate (generic) functions to indexed datatypesI Consumers (like show) are easyI Producers (like read, or our parse) are trickier!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 23 / 46

Page 41: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Parsing chord sequences III

Two challenges to triviality:

1. How to do generic programming with indexed datatypesI Current libraries can’t do it (Uniplate, SYB, GHC.Generics, etc.)I So we came up with oneI . . . which kind of works.

2. How to instantiate (generic) functions to indexed datatypesI Consumers (like show) are easyI Producers (like read, or our parse) are trickier!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 23 / 46

Page 42: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Diversion 1: GP for indexed datatypes I

Indexed datatypes are actually “just” existential quantification togetherwith equality constraints:

data Nat = Ze | Su Nat

data Vec (η :: Nat) (α :: ?) whereNil :: Vec Ze αCons :: α→ Vec η α→ Vec (Su η) α

This can also be written as such:

data Vec (η :: Nat) (α :: ?) =η ∼ Ze ⇒ Nil

| ∀µ :: Nat.η ∼ Su µ⇒ Cons α (Vec µ α)

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 24 / 46

Page 43: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Diversion 1: GP for indexed datatypes II

Handling equality constraints generically is easy: we just add a typeequality operator to the generic representation.Existential quantification is much harder. My current solution usesskolemization; alternatives are welcome!

type instance Rep (Vec η α) = -- can’t place a forall here!(η :∼: Ze) :×: U

:+: (η :∼: Su (X η)) :×: α :×: Vec (X η) α

type family X (σ :: κ)type instance X (Su σ) = σ

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 25 / 46

Page 44: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Diversion 2: instantiating functions to indexed datatypes IIndexed datatypes need one instance per type at which any of theirconstructors is indexed at, and possibly one more in case someconstructors do not constrain the index. E.g.:

data Index = I1 | I2

data Data (ι :: Index) whereD1 :: Data I1D2 :: Data I2D3 :: Data ι

instance Read (Data I1) whereread "D1" = D1

instance Read (Data I2) whereread "D2" = D2

instance Read (Data ι) whereread "D3" = D3

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 26 / 46

Page 45: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Diversion 2: instantiating functions to indexed datatypes IIndexed datatypes need one instance per type at which any of theirconstructors is indexed at, and possibly one more in case someconstructors do not constrain the index. E.g.:

data Index = I1 | I2

data Data (ι :: Index) whereD1 :: Data I1D2 :: Data I2D3 :: Data ι

instance Read (Data I1) whereread "D1" = D1

read "D3" = D3 -- don’t forget these!

instance Read (Data I2) whereread "D2" = D2

read "D3" = D3 -- don’t forget these!

instance Read (Data ι) whereread "D3" = D3

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 26 / 46

Page 46: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Diversion 2: instantiating functions to indexed datatypes II

I think we should:

1. Add support for indexed datatypes in GHC.Generics;

2. Let users use deriving for any class which has a genericimplementation;

3. Change the semantics of deriving for indexed datatypes (the currentsemantics is pretty much undefined or failure right now, anyway).

Then we’ll finally be able to write:

data Vec (η :: Nat) (α :: ?) whereNil :: Vec Ze αCons :: α→ Vec η α→ Vec (Su η) αderiving (Show,Read)

(And I’ll be able to do the same for all of my harmony model, too!)

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 27 / 46

Page 47: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Diversion 2: instantiating functions to indexed datatypes II

I think we should:

1. Add support for indexed datatypes in GHC.Generics;

2. Let users use deriving for any class which has a genericimplementation;

3. Change the semantics of deriving for indexed datatypes (the currentsemantics is pretty much undefined or failure right now, anyway).

Then we’ll finally be able to write:

data Vec (η :: Nat) (α :: ?) whereNil :: Vec Ze αCons :: α→ Vec η α→ Vec (Su η) αderiving (Show,Read)

(And I’ll be able to do the same for all of my harmony model, too!)

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 27 / 46

Page 48: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Diversion 2: instantiating functions to indexed datatypes II

I think we should:

1. Add support for indexed datatypes in GHC.Generics;

2. Let users use deriving for any class which has a genericimplementation;

3. Change the semantics of deriving for indexed datatypes (the currentsemantics is pretty much undefined or failure right now, anyway).

Then we’ll finally be able to write:

data Vec (η :: Nat) (α :: ?) whereNil :: Vec Ze αCons :: α→ Vec η α→ Vec (Su η) αderiving (Show,Read)

(And I’ll be able to do the same for all of my harmony model, too!)

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 27 / 46

Page 49: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Diversion 2: instantiating functions to indexed datatypes II

I think we should:

1. Add support for indexed datatypes in GHC.Generics;

2. Let users use deriving for any class which has a genericimplementation;

3. Change the semantics of deriving for indexed datatypes (the currentsemantics is pretty much undefined or failure right now, anyway).

Then we’ll finally be able to write:

data Vec (η :: Nat) (α :: ?) whereNil :: Vec Ze αCons :: α→ Vec η α→ Vec (Su η) αderiving (Show,Read)

(And I’ll be able to do the same for all of my harmony model, too!)

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 27 / 46

Page 50: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Table of Contents

Harmony

A functional model of harmony

Harmony analysis

Harmonic similarity

Music generation

Chord recognition: Chordify

Crowd-sourcing chord edits

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 28 / 46

Page 51: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Application: harmonic similarity

I A practical application of a harmony model is to estimate harmonicsimilarity between songs

I The more similar the trees, the more similar the harmony

I We don’t want to write a diff algorithm for our complicated model;we get it automatically by using a generic diff

I The generic diff is a type-safe tree-diff algorithm, part of a student’sMSc work at Utrecht University

I Generic, thus working for any model, and independent of changes tothe model

I Once again, we’re giving a new meaning to programming languagetechniques by applying them to musical harmony!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 29 / 46

Page 52: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Table of Contents

Harmony

A functional model of harmony

Harmony analysis

Harmonic similarity

Music generation

Chord recognition: Chordify

Crowd-sourcing chord edits

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 30 / 46

Page 53: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Application: automatic harmonisation of melodies

Another practical application of a harmony model is to help selectinggood harmonisations (chord sequences) for a given melody:

!"

" !""IV

!

!"

"""III

!V

# $% $

"

""

"

"""V

"I

"

"""III

"

"""IV

"

"""III

"

"""II

Music engraving by LilyPond 2.16.2—www.lilypond.org

We generate candidate chord sequences, parse them with the harmonymodel, and select the one with the least errors.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 31 / 46

Page 54: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Visualising harmonic structure

Piece

Phrase

Ton

I: Maj

C: Maj

Dom

Dom

V: Dom7

G: Dom7

II: Dom7

D: Dom7

Sub

IV: Maj

F: Maj

III: Min

E: Min

Ton

I: Maj

C: Maj

You can see this tree as having been produced by taking the chords ingreen as input. . .

or the chords might have been dictated by the structure!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 32 / 46

Page 55: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Generating harmonic structure

Piece

Phrase

Ton

I: Maj

C: Maj

Dom

Dom

V: Dom7

G: Dom7

II: Dom7

D: Dom7

Sub

IV: Maj

F: Maj

III: Min

E: Min

Ton

I: Maj

C: Maj

You can see this tree as having been produced by taking the chords ingreen as input. . . or the chords might have been dictated by the structure!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 32 / 46

Page 56: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Generating harmony

Now that we have a datatype representing harmony sequences, how dowe generate a sequence of chords?

QuickCheck! We give Arbitrary instances for the datatypes in our model.

. . . but we don’t want to do this by hand, for every datatype, and to haveto adapt the instances every time we change the model. . . but genericprogramming saves us once again:

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 33 / 46

Page 57: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Generating harmony

Now that we have a datatype representing harmony sequences, how dowe generate a sequence of chords?

QuickCheck! We give Arbitrary instances for the datatypes in our model.

. . . but we don’t want to do this by hand, for every datatype, and to haveto adapt the instances every time we change the model. . . but genericprogramming saves us once again:

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 33 / 46

Page 58: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Generating harmony

Now that we have a datatype representing harmony sequences, how dowe generate a sequence of chords?

QuickCheck! We give Arbitrary instances for the datatypes in our model.

. . . but we don’t want to do this by hand, for every datatype, and to haveto adapt the instances every time we change the model. . . but genericprogramming saves us once again:

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 33 / 46

Page 59: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Generating harmony

Now that we have a datatype representing harmony sequences, how dowe generate a sequence of chords?

QuickCheck! We give Arbitrary instances for the datatypes in our model.

. . . but we don’t want to do this by hand, for every datatype, and to haveto adapt the instances every time we change the model. . . but genericprogramming saves us once again:

gen :: ∀α.(Representable α,Generate (Rep α))⇒ Gen α

(You might recall my talk at the last IFIP meeting. . . )

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 33 / 46

Page 60: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Generating harmony

Now that we have a datatype representing harmony sequences, how dowe generate a sequence of chords?

QuickCheck! We give Arbitrary instances for the datatypes in our model.

. . . but we don’t want to do this by hand, for every datatype, and to haveto adapt the instances every time we change the model. . . but genericprogramming saves us once again:

gen :: ∀α.(Representable α,Generate (Rep α))⇒ [ (String,Int) ]→ Gen α

(You might recall my talk at the last IFIP meeting. . . )

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 33 / 46

Page 61: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Examples of harmony generation

testGen :: Gen (Phrase MajMode)testGen = gen [("Dom_IV-V", 3), ("Dom_II-V", 4)]

example :: IO ()example = let k = Key (Note \ C) MajMode

in sample′ testGen>>= mapM (printOnKey k)

> example[C: Maj,D: Dom7,G: Dom7,C: Maj][C: Maj,G: Dom7,C: Maj][C: Maj,E: Min,F: Maj,G: Maj,C: Maj][C: Maj,E: Min,F: Maj,D: Dom7,G: Dom7,C: Maj][C: Maj,D: Min,E: Min,F: Maj,D: Dom7,G: Dom7,C: Maj]

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 34 / 46

Page 62: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Examples of harmony generation

testGen :: Gen (Phrase MajMode)testGen = gen [("Dom_IV-V", 3), ("Dom_II-V", 4)]

example :: IO ()example = let k = Key (Note \ C) MajMode

in sample′ testGen>>= mapM (printOnKey k)

> example[C: Maj,D: Dom7,G: Dom7,C: Maj][C: Maj,G: Dom7,C: Maj][C: Maj,E: Min,F: Maj,G: Maj,C: Maj][C: Maj,E: Min,F: Maj,D: Dom7,G: Dom7,C: Maj][C: Maj,D: Min,E: Min,F: Maj,D: Dom7,G: Dom7,C: Maj]

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 34 / 46

Page 63: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Generating a melody for a given harmony

Harmonies without a melody are boring. I don’t (yet) have a functionalmodel of melody, so let’s do something more mundane:

1. Generate a list of candidate melody notes per chord;

2. Refine the candidates by filtering out obviously bad candidates;

3. Pick one focal candidate melody note per chord;

4. Embellish the candidate notes to produce a final melody.

These four steps combine naturally using plain monadic bind:

melody :: Key→ State MyState Songmelody k = genCandidates>>= refine>>= pickOne>>= embellish

>>= return ◦ Song k

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 35 / 46

Page 64: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Generating a melody for a given harmony

Harmonies without a melody are boring. I don’t (yet) have a functionalmodel of melody, so let’s do something more mundane:

1. Generate a list of candidate melody notes per chord;

2. Refine the candidates by filtering out obviously bad candidates;

3. Pick one focal candidate melody note per chord;

4. Embellish the candidate notes to produce a final melody.

These four steps combine naturally using plain monadic bind:

melody :: Key→ State MyState Songmelody k = genCandidates>>= refine>>= pickOne>>= embellish

>>= return ◦ Song k

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 35 / 46

Page 65: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Example I

&

?

œ œ œ œ œ œ œ œ œœ œ œ œ œ œ œ œ œ œ œ

œœœ

œœœ

œœœ

œœœœ#

œœœœ

n

œœœ

Phrase

Ton

I: Maj

C: Maj

Dom

Dom

V: Dom7

G: Dom7

II: Dom7

D: Dom7

Sub

IV: Maj

F: Maj

III: Min

E: Min

Ton

I: Maj

C: Maj

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 36 / 46

Page 66: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Example II

&

#

?#

œ œ œ œ œ œ œ œ œ œ œ œ œ

œœœ

œœœ

œœœ

œœœœ##

œœœœ

n

#œœœ

Phrase

Ton

I: Min

E: Min

Dom

Dom

Dom

V: Dom7

B: Dom7

II: Dom7

F]: Dom7

Sub

IV: Min

A: Min

Sub

IV: Min

A: Min

Ton

I: Min

E: Min

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 37 / 46

Page 67: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Table of Contents

Harmony

A functional model of harmony

Harmony analysis

Harmonic similarity

Music generation

Chord recognition: Chordify

Crowd-sourcing chord edits

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 38 / 46

Page 68: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Chord recognition: Chordify

Yet another practical application of a harmony model is to improve chordrecognition from audio sources.

0.92 C 0.96 EChord candidates 0.94 Gm 0.97 C

1.00 C 1.00 G 1.00 EmBeat number 1 2 3

How to pick the right chord from the chord candidate list? Ask theharmony model which one fits best.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 39 / 46

Page 69: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Demo: Chordify

Demo:

http://chordify.net

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 40 / 46

Page 70: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Chordify: architecture

I FrontendI Reads user input, such as YouTube/Soundcloud/Deezer links, or filesI Extracts audioI Calls the backend to obtain the chords for the audioI Displays the result to the userI Implements a queueing system, and library functionalityI Uses PHP, JavaScript, MongoDB

I BackendI Takes an audio file as input, analyses it, extracts the chordsI The chord extraction code uses GADTs, type families, generic

programming (see the HarmTrace package on Hackage)I Performs PDF and MIDI export (using LilyPond)I Uses Haskell, SoX, sonic annotator, and is mostly open source

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 41 / 46

Page 71: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Chordify: architecture

I FrontendI Reads user input, such as YouTube/Soundcloud/Deezer links, or filesI Extracts audioI Calls the backend to obtain the chords for the audioI Displays the result to the userI Implements a queueing system, and library functionalityI Uses PHP, JavaScript, MongoDB

I BackendI Takes an audio file as input, analyses it, extracts the chordsI The chord extraction code uses GADTs, type families, generic

programming (see the HarmTrace package on Hackage)I Performs PDF and MIDI export (using LilyPond)I Uses Haskell, SoX, sonic annotator, and is mostly open source

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 41 / 46

Page 72: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Chordify: numbers

I Online since January 2013

I Top countries: US, UK, Germany, Indonesia, Canada

I Views: 3M+ (monthly)

I Chordified songs: 1.5M+

I Registered users: 200K+

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 42 / 46

Page 73: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Table of Contents

Harmony

A functional model of harmony

Harmony analysis

Harmonic similarity

Music generation

Chord recognition: Chordify

Crowd-sourcing chord edits

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 43 / 46

Page 74: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Merging chord edits I

The next major feature planned for Chordify is to allow users to edit theautomatically-recognised chord sequences.

Two ways to make the edit process collaborative:

1. Wiki-style: your edits are visible to everyone;

2. You edit only your own version; the rest of the world still gets theautomatically-recognised chords.

We don’t like (1); too prone to abuse. But (2) is not collaborative atall. . . unless we automatically improve the “automatically-recognised”default chords with user edits!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 44 / 46

Page 75: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Merging chord edits I

The next major feature planned for Chordify is to allow users to edit theautomatically-recognised chord sequences.

Two ways to make the edit process collaborative:

1. Wiki-style: your edits are visible to everyone;

2. You edit only your own version; the rest of the world still gets theautomatically-recognised chords.

We don’t like (1); too prone to abuse. But (2) is not collaborative atall. . . unless we automatically improve the “automatically-recognised”default chords with user edits!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 44 / 46

Page 76: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Merging chord edits I

The next major feature planned for Chordify is to allow users to edit theautomatically-recognised chord sequences.

Two ways to make the edit process collaborative:

1. Wiki-style: your edits are visible to everyone;

2. You edit only your own version; the rest of the world still gets theautomatically-recognised chords.

We don’t like (1); too prone to abuse. But (2) is not collaborative atall. . . unless we automatically improve the “automatically-recognised”default chords with user edits!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 44 / 46

Page 77: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Merging chord edits II

How will we do this? I don’t know! But here are some ideas:

I Use metrics such as:I Regularity (e.g. do the chords change on strong metrical positions?)I Repetition (music always contains repetition)I How well it fits in the harmony model (e.g. number of parsing errors)I User reputation

I Detect concordance among edits by several users;

I . . . and hopefully we’ll get a PhD student to work a bit on this.

Ultimate goal: have a public database of high-quality human-annotatedsongs from public sources (Youtube/SoundCloud)—this is lacking, andcould be quite helpful to improve automatic chord estimation.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 45 / 46

Page 78: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Merging chord edits II

How will we do this? I don’t know! But here are some ideas:

I Use metrics such as:I Regularity (e.g. do the chords change on strong metrical positions?)I Repetition (music always contains repetition)I How well it fits in the harmony model (e.g. number of parsing errors)I User reputation

I Detect concordance among edits by several users;

I . . . and hopefully we’ll get a PhD student to work a bit on this.

Ultimate goal: have a public database of high-quality human-annotatedsongs from public sources (Youtube/SoundCloud)—this is lacking, andcould be quite helpful to improve automatic chord estimation.

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 45 / 46

Page 79: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Summary

Musical modelling with Haskell:

I A model for musical harmony as a Haskell datatype

I Makes use of several advanced functional programming techniques,such as generic programming, GADTs, and type families

I Analysing harmony—using error-correcting parsers

I Finding cover songs—with a generic diff

I Generating harmonies—with QuickCheck

I Recognising harmony from audio sources

I The synergy between two different CS fields, and the advantages oftransporting academic research into industry

Thank you for your time!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 46 / 46

Page 80: Functional Modelling of Musical Harmony And Its …dreixel.net/research/pdf/fmmhaia_pres_ifip14.pdf · Functional Modelling of Musical Harmony And Its Applications Jos e Pedro Magalh~aes

Summary

Musical modelling with Haskell:

I A model for musical harmony as a Haskell datatype

I Makes use of several advanced functional programming techniques,such as generic programming, GADTs, and type families

I Analysing harmony—using error-correcting parsers

I Finding cover songs—with a generic diff

I Generating harmonies—with QuickCheck

I Recognising harmony from audio sources

I The synergy between two different CS fields, and the advantages oftransporting academic research into industry

Thank you for your time!

Jose Pedro Magalhaes Functional Modelling of Musical Harmony, IFIP 2.1 Vermont 46 / 46