Harmony Analysis - International Audio Laboratories Erlangen

Harmony Analysis

Music Information Retrieval

Christof Weiß, Frank Zalkow, Meinard Müller

Workshop HfM Karlsruhe

International Audio Laboratories Erlangen

[email protected]@audiolabs-erlangen.de

[email protected]

Book: Fundamentals of Music Processing

Meinard MüllerFundamentals of Music ProcessingAudio, Analysis, Algorithms, Applications483 p., 249 illus., hardcoverISBN: 978-3-319-21944-8Springer, 2015

Accompanying website: www.music-processing.de

Book: Fundamentals of Music Processing

Meinard MüllerFundamentals of Music ProcessingAudio, Analysis, Algorithms, Applications483 p., 249 illus., hardcoverISBN: 978-3-319-21944-8Springer, 2015

Accompanying website: www.music-processing.de

5.1 Basic Theory of Harmony5.2 Template-Based Chord Recognition5.3 HMM-Based Chord Recognition5.4 Further Notes

In Chapter 5, we consider the problem of analyzing harmonic properties of apiece of music by determining a descriptive progression of chords from a givenaudio recording. We take this opportunity to first discuss some basic theory ofharmony including concepts such as intervals, chords, and scales. Then,motivated by the automated chord recognition scenario, we introducetemplate-based matching procedures and hidden Markov models—a conceptof central importance for the analysis of temporal patterns in time-dependentdata streams including speech, gestures, and music.

Chapter 5: Chord Recognition

Dissertation: Tonality-Based Style Analysis

Christof WeißComputational Methods for Tonality-Based Style Analysis of Classical Music Audio RecordingsDissertation, Ilmenau University of Technology, 2017

Chapter 5: Analysis Methods for Key and Scale StructuresChapter 6: Design of Tonal Features

Chromatic circle Shepard’s helix of pitch

Recall: Chroma Features Human perception of pitch is periodic Two components: tone height (octave) and chroma (pitch class)

Time (seconds)

Chr

oma

Recall: Chroma Features

Salie

nce

/ Li

kelih

ood

Recall: Chroma Representations

Salie

nce

/ Lik

elih

ood

L. van Beethoven,Fidelio, Overture,Slovak Philharmonic

Recall: Chroma Representations Orchestra

Piano

L. van Beethoven,Fidelio, Overture,Slovak Philharmonic

Fidelio, Overture,arr. Alexander ZemlinskyM. Namekawa, D.R. Davies, piano four hands

Tonal Structures

ChordsCM GM7 Am

Global key

Local keyC major G major C major

Global key detection

Chord recognition

Music transcriptionNote level

Segment level

Chord level

Movement level C major

MelodyMiddle voices

Bass line

Local key detection

Western music (and most other music): Different aspects of harmony Referring to different time scales

Tonal Structures

ChordsCM GM7 Am

Global key



Chord recognition


Segment level

Chord level


MelodyMiddle voices

Bass line

Local key detection

Western music (and most other music): Different aspects of harmony Referring to different time scales

Chord Recognition

Source: www.ultimate-guitar.com

Time (seconds)

Chord Recognition

Chord Recognition

Chord Recognition

C G

Audiorepresentation

Prefiltering▪ Compression▪ Overtones▪ Smoothing

▪ Smoothing▪ Transition▪ HMM

Chromarepresentation

Patternmatching

Recognitionresult

Postfiltering

Majortriads

Minortriads

Chord Recognition: Basics Musical chord: Group of three or more notes Combination of three or more tones which sound simultaneously Types: triads (major, minor, diminished, augmented), seventh chords… Here: focus on major and minor triads

C

→ C Major (C)

Chord Recognition: Basics Musical chord: Group of three or more notes Combination of three or more tones which sound simultaneously Types: triads (major, minor, diminished, augmented), seventh chords… Here: focus on major and minor triads

Enharmonic equivalence: 12 different root notes possible → 24 chords

C Major (C)

C Minor (Cm)

Chord Recognition: BasicsChords appear in different forms: Inversions

Different voicings

Harmonic figuration: Broken chords (arpeggio)

Melodic figuration: Different melody note (suspension, passing tone, …) Further: Additional notes, incomplete chords

Chord Recognition: Basics

B

A

G

FE

D

C

G♯/A♭

D♯/E♭

C♯/D♭

A♯/B♭

F♯/G♭

C D ♭ D E ♭ E F G ♭ G A ♭ A B ♭ B

Templates: Major Triads


B

A

G

FE

D

C

G♯/A♭

D♯/E♭

C♯/D♭

A♯/B♭

F♯/G♭

C D ♭ D E ♭ E F G ♭ G A ♭ A B ♭ B

Templates: Major Triads


B

A

G

FE

D

C

G♯/A♭

D♯/E♭

C♯/D♭

A♯/B♭

F♯/G♭

Cm C♯m Dm E ♭m Em Fm F♯m Gm G♯m Am B ♭m Bm

Templates: Minor Triads

Chroma vectorfor each audio frame

24 chord templates(12 major, 12 minor)

Compute for each frame thesimilarity of the chroma vector

to the 24 templates

B

A

G

F

E

D

C

G♯

D♯

C♯

A♯

F♯

C C♯ D … Cm C♯m Dm

0 0 0 … 0 0 0 …

0 0 0 … 0 0 0 …

0 0 1 … 0 0 1 …

0 1 0 … 0 1 0 …

1 0 0 … 1 0 0 …

0 0 1 … 0 0 0 …

0 1 0 … 0 0 1 …

1 0 0 … 0 1 0 …

0 0 0 … 1 0 0 …

0 0 1 … 0 0 1 …

0 1 0 … 0 1 0 …

1 0 0 … 1 0 0 …

…

Chord Recognition: Template Matching

Chord Recognition: Template Matching Similarity measure: Cosine similarity (inner product of normalized

vectors)

Chord template:

Chroma vector:

Similarity measure:

Cho

rdC

hrom

a

Chord Recognition: Template Matching

Time (seconds)

Chord Recognition: Label Assignment

Assign to each frame the chord labelof the template that maximizes the

similarity to the chroma vector

B

A

G

F

E

D

C

G♯

D♯

C♯

A♯

F♯

C C♯ D … Cm C♯m Dm

0 0 0 … 0 0 0 …

0 0 0 … 0 0 0 …

0 0 1 … 0 0 1 …

0 1 0 … 0 1 0 …

1 0 0 … 1 0 0 …

0 0 1 … 0 0 0 …

0 1 0 … 0 0 1 …

1 0 0 … 0 1 0 …

0 0 0 … 1 0 0 …

0 0 1 … 0 0 1 …

0 1 0 … 0 1 0 …

1 0 0 … 1 0 0 …

…Chroma vector

for each audio frame24 chord templates(12 major, 12 minor)

Compute for each frame thesimilarity of the chroma vector

to the 24 templates

Chord Recognition: Label Assignment

Cho

rdC

hord

Time (seconds)

Chord Recognition: Evaluation

Time (seconds)

C G Am F C G F C

Chord Recognition: Evaluation “No-Chord” annotations: not every frame labeled

Different evaluation measures: Precision:

Recall:

F-Measure (balances precision and recall):

Without “No-Chord” label:

𝑃#TP

#TP #FP

𝑅#TP

#TP #FN

𝐹2 ⋅ 𝑃 ⋅ 𝑅𝑃 𝑅

𝑃 𝑅 𝐹

Chord Recognition: Smoothing Apply average filter of length 𝐿 ∈ ℕ:

Time (seconds)

Chord Recognition: Smoothing Apply average filter of length 𝐿 ∈ ℕ:

Time (seconds)

Chord Recognition: Smoothing Evaluation on all Beatles songs

0.8

0.75

0.7

0.65

0.6

0.55

0.5

0.45

0.4

0.35

Smoothing length

F-m

easu

re

Binary templates

1 5 9 13 17 21 25

Markov Chains Probabilistic model for sequential data Markov property: Next state only depends on current state

(no “memory”) Consist of: Set of states (hidden)

State transition probabilities

Initial state probabilitiesC

FG

0.8

0.60.7

0.2

0.1

0.3

0.10.1

0.1

Markov ChainsNotation:

C

FG

0.8

0.60.7

0.2

0.1

0.3

0.10.1

0.1

A α1 α2 α3

α1 𝑎 𝑎 𝑎α2 𝑎 𝑎 𝑎α3 𝑎 𝑎 𝑎


Initial state probabilities

C α1 α2 α3

𝑐 𝑐 𝑐

𝑎

𝑐

States 𝛼 for 𝑖 ∈ 1: 𝐼

𝛽 for 𝑘 ∈ 1: 𝐾

Markov Chains Application examples: Compute probability of a sequence using given a model (evaluation) Compare two sequences using a given model Evaluate a sequence with two different models (classification)

C

FG

0.8

0.60.7

0.2

0.1

0.3

0.10.1

0.1

Hidden Markov Models

C

FG

0.8

0.60.7

0.2

0.1

0.3

0.10.1

0.1

States as hidden variables

Consist of: Set of states (hidden)



Hidden Markov Models

C

FG

0.8

0.60.7

0.2

0.1

0.3

0.10.1

0.1

States as hidden variables




Observations (visible)

Hidden Markov Models States as hidden variables





Emission probabilities

0.7

0.1

0.9 0.2

0.3

0.8

C

FG

0.8

0.60.7

0.2

0.1

0.3

0.10.1

0.1

0

00

Hidden Markov ModelsNotation:

0.7

0.1

0.9 0.2

0.3

0.8

C

FG

0.8

0.60.7

0.2

0.1

0.3

0.10.1

0.1

0

00

A α1 α2 α3


B β1 β2 β3

α1 𝑏 𝑏 𝑏α2 𝑏 𝑏 𝑏α3 𝑏 𝑏 𝑏




C α1 α2 α3

𝑐 𝑐 𝑐

𝑎

𝑏

𝑐

States 𝛼 for 𝑖 ∈ 1: 𝐼

Observation symbols 𝛽 for 𝑘 ∈ 1: 𝐾

Markov Chains Analogon: the student‘s life Consists of: Set of states (hidden)


Initial state probabilities0.5

0.30.4

0.2

0.2

codingsleep

eatingsocialactivity

0.7

0.1

0.2

0.1 0.10.3 0

0.1

0.2

0.4

0.2

Hidden Markov Models Analogon: the student‘s life Consists of: Set of states (hidden)





0.5

0.30.4

0.2

0.2

codingsleep

eatingsocialactivity

0.7

0.1

0.2

0.1 0.10.3 0

0.1

0.2

0.4

0.2

0.9

Smell

NoiseLight

0.5

0.3 0.4

0.3

0.7

Hidden Markov Models Only observation sequence is visible!Different algorithmic problems: Evaluation problem Given: observation sequence and model Calculate how well the model matches the sequence

Uncovering problem: Given: observation sequence and model Find: optimal hidden state sequence

Estimation problem („training“ the HMM): Given: observation sequence Find: model parameters Baum-Welch algorithm (Expectation-Maximization)

Uncovering problem Given: observation sequence 𝑂 𝑜 , … , 𝑜 of length 𝑁 ∈ ℕ and

HMM 𝛩 (model parameters) Find: optimal hidden state sequence 𝑆∗ 𝑠∗, … , 𝑠∗

Corresponds to chord estimation task!

Observation sequence

β1 β3 β1 β3 β3 β2

𝑂 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜


β1 β3 β1 β3 β3 β2

𝑂 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜

α1 α1 α1 α3 α3 α1

Hidden state sequence 𝑆∗ 𝑠∗ , 𝑠∗ , 𝑠∗ , 𝑠∗ , 𝑠∗ , 𝑠∗


HMM 𝛩 (model parameters) Find: optimal hidden state sequence Corresponds to chord estimation task!


β1 β3 β1 β3 β3 β2

𝑂 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜 , 𝑜

C C C G G C

Hidden state sequence 𝑆∗ 𝑠∗ , 𝑠∗ , 𝑠∗ , 𝑠∗ , 𝑠∗ , 𝑠∗


HMM 𝛩 (model parameters) Find: optimal hidden state sequence Corresponds to chord estimation task!

Uncovering problem Optimal hidden state sequence? “Best explains” given observation sequence 𝑂 Maximizes probability 𝑃 𝑂, 𝑆 Θ

Straight-forward computation (naive approach): Compute probability for each possible sequence 𝑆

Number of possible sequences of length 𝑁 (𝐼 number of states):

Prob∗ max 𝑃 𝑂, 𝑆 Θ

𝑆∗ argmax 𝑃 𝑂, 𝑆 Θ

𝐼 · 𝐼 ·… · 𝐼 𝐼

𝑁 factorscomputationally infeasible!

Viterbi Algorithm Based on dynamic programming (similar to DTW) Idea: Recursive computation from subproblems Use truncated versions of observation sequence

Define 𝐃 𝑖, 𝑛 as the highest probability along a single state sequence𝑠 , … , 𝑠 that ends in state 𝑠 𝛼

Then, our solution is the state sequence yielding

Prob∗ max∈ :

𝐃 𝑖, 𝑁

𝑂 1: 𝑛 ≔ 𝑜 , … , 𝑜 , length 𝑛 ∈ 1: 𝑁

𝐃 𝑖, 𝑛 max,… ,

𝑃 𝑂 1: 𝑛 , 𝑠 , … , 𝑠 , 𝑠 𝛼 Θ

Viterbi Algorithm 𝐃: matrix of size 𝐼 𝑁

Recursive computation of 𝐃 𝑖, 𝑛 along the column index 𝑛

Initialization: 𝑛 1

Truncated observation sequence: 𝑂 1 𝑜 Current observation: 𝑜 𝛽

𝐃 𝑖, 1 𝑐 ⋅ 𝑏 for some 𝑖 ∈ 1: 𝐼



Recursion: 𝑛 ∈ 2: 𝑁

Truncated observation sequence: 𝑂 1: 𝑛 𝑜 , … , 𝑜 Last observation: 𝑜 𝛽

𝐃 𝑖, 𝑛 𝑏 ⋅ 𝑎 ∗ ⋅ 𝑃 𝑂 1: 𝑛 1 , 𝑠 , … , 𝑠 𝛼 ∗ Θ for 𝑖 ∈ 1: 𝐼

𝐃 𝑖, 𝑛 𝑏 ⋅ 𝑎 ∗ ⋅ 𝐃 𝑗∗, 𝑛 1

must be maximal!



Recursion: 𝑛 ∈ 2: 𝑁

Truncated observation sequence: 𝑂 1: 𝑛 𝑜 , … , 𝑜 Last observation: 𝑜 𝛽

𝐃 𝑖, 𝑛 𝑏 ⋅ 𝑎 ∗ ⋅ 𝑃 𝑂 1: 𝑛 1 , 𝑠 , … , 𝑠 𝛼 ∗ Θ for 𝑖 ∈ 1: 𝐼

𝐃 𝑖, 𝑛 𝑏 ⋅ 𝑎 ∗ ⋅ 𝐃 𝑗∗, 𝑛 1

must be maximal!

must be maximal (best index 𝑗∗)

𝐃 𝑖, 𝑛 𝑏 ⋅ max∈ :

𝑎 ⋅ 𝐃 𝑗, 𝑛 1

Viterbi Algorithm 𝐃 given – find optimal state sequence 𝑆∗ 𝑠∗, … , 𝑠∗ ≔ 𝛼 , … , 𝛼

Backtracking procedure (reverse order) Last element: 𝑛 𝑁 Optimal state: 𝛼

𝑖 argmax∈ :

𝐃 𝑗, 𝑁


Backtracking procedure (reverse order) Further elements: 𝑛 𝑁 1, 𝑁 2, … , 1 Optimal state: 𝛼

𝑖 argmax∈ :

𝑎 ⋅ 𝐃 𝑖, 𝑛


Backtracking procedure (reverse order) Further elements: 𝑛 𝑁 1, 𝑁 2, … , 1 Optimal state: 𝛼

Simplification of backtracking: Keep track of maximizing index 𝑗 in

Define 𝐼 𝑁 1 matrix 𝐄:

𝑖 argmax∈ :

𝑎 ⋅ 𝐃 𝑖, 𝑛


𝑎 ⋅ 𝐃 𝑗, 𝑛 1

𝐄 𝑖, 𝑛 1 argmax∈ :

𝑎 ⋅ 𝐃 𝑗, 𝑛 1

Viterbi AlgorithmSummary

1

1

2

3

4

5

6

7

8

…𝑐𝑖 𝑏𝑖

𝐷 𝑖, 1𝑘1

States𝑖 ∈ 1: 𝐼

Sequence index 𝑛 ∈ 1: 𝑁

Initialization


1

1

2

3

4

5

6

7

8

…… …

……… …

…

…

…

…

n – 1 n

… …𝑐𝑖 𝑏𝑖

𝐷 𝑖, 1𝑘1



Initialization Recursion


1

1

2

3

4

5

6

7

8

…… …

……… …

…

…

…

…

n – 1 n

𝑏𝑖𝑘𝑛

𝐷 𝑖, 𝑛


𝐷 𝑖, 1𝑘1





1

1

2

3

4

5

6

7

8

…… …

……… …

…

…

…

…

n – 1 n

𝑎𝑗𝑖𝑏𝑖𝑘𝑛

𝐷 𝑗, 𝑛 1

𝐷 𝑖, 𝑛


𝐷 𝑖, 1𝑘1




…… …

… …

……

…

…

…

…

…… …

…

…

…


1

1

2

3

4

5

6

7

8

n – 1 n N


𝐷 𝑗, 𝑛 1

𝐷 𝑖, 𝑛

𝐷 𝑖, 𝑁


𝐷 𝑖, 1𝑘1



Initialization Recursion Termination

…… …

… …

……

…

…

…

…

…… …

…

…

…


1

1

2

3

4

5

6

7

8

n – 1 n N


𝐷 𝑗, 𝑛 1

𝐷 𝑖, 𝑛

𝐷 𝑖, 𝑁


𝐷 𝑖, 1𝑘1



Initialization Recursion Termination

Backtrackingmatrix 𝐄

…… …

…

……

Viterbi AlgorithmComputational Complexity

n – 1 n


𝐷 𝑗, 𝑛 1

𝐷 𝑖, 𝑛



Recursion𝐼 states 𝐼 states

Per recursion step:𝐼 · 𝐼

Total recursion:𝐼 · 𝑁


Viterbi Algorithm: Example

A α1 α2 α3


B β1 β2 β3

α1 𝑏 𝑏 𝑏α2 𝑏 𝑏 𝑏α3 𝑏 𝑏 𝑏

State transition probabilities Emission probabilities Initial state probabilities

C α1 α2 α3

𝑐 𝑐 𝑐

𝑎 𝑏 𝑐

States𝛼 for 𝑖 ∈ 1: 𝐼

Observation symbols𝛽 for 𝑘 ∈ 1: 𝐾

HMM:


A α1 α2 α3

α1 0.8 0.1 0.1α2 0.2 0.7 0.1α3 0.1 0.3 0.6

B β1 β2 β3

α1 0.7 0 0.3α2 0.1 0.9 0α3 0 0.2 0.8

C α1 α2 α3

0.6 0.2 0.2

State transition probabilities Emission probabilities Initial state probabilities𝑎 𝑏 𝑐



HMM:


A α1 α2 α3

α1 0.8 0.1 0.1α2 0.2 0.7 0.1α3 0.1 0.3 0.6

B β1 β2 β3

α1 0.7 0 0.3α2 0.1 0.9 0α3 0 0.2 0.8

C α1 α2 α3

0.6 0.2 0.2




HMM:


β1 β3 β1 β3 β3 β2

O = (o1,o2,o3,o4,o5,o6)

Input


A α1 α2 α3

α1 0.8 0.1 0.1α2 0.2 0.7 0.1α3 0.1 0.3 0.6

B β1 β2 β3

α1 0.7 0 0.3α2 0.1 0.9 0α3 0 0.2 0.8

C α1 α2 α3

0.6 0.2 0.2




HMM:

D o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2

α1

α2

α3

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1

α2

α3


β1 β3 β1 β3 β3 β2

O = (o1,o2,o3,o4,o5,o6)

Input Viterbi algorithm


A α1 α2 α3

α1 0.8 0.1 0.1α2 0.2 0.7 0.1α3 0.1 0.3 0.6

B β1 β2 β3

α1 0.7 0 0.3α2 0.1 0.9 0α3 0 0.2 0.8

C α1 α2 α3

0.6 0.2 0.2




HMM:

D o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2

α1

α2

α3

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1

α2

α3

Viterbi algorithm

𝐃 𝑖, 1 𝑐 ⋅ 𝑏

Initialization

β1 β3 β1 β3 β3 β2

Inputo1 o2 o3 o4 o5 o6


A α1 α2 α3

α1 0.8 0.1 0.1α2 0.2 0.7 0.1α3 0.1 0.3 0.6

B β1 β2 β3

α1 0.7 0 0.3α2 0.1 0.9 0α3 0 0.2 0.8

C α1 α2 α3

0.6 0.2 0.2




HMM:

D o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2

α1 0.4200α2 0.0200α3 0

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1

α2

α3

Viterbi algorithm


𝑎 ⋅ 𝐃 𝑗, 𝑛 1


𝑎 ⋅ 𝐃 𝑗, 𝑛 1

𝐃 𝑖, 1 𝑐 ⋅ 𝑏

Initialization

Recursion

β1 β3 β1 β3 β3 β2


D o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2

α1 0.4200 0.1008 α2 0.0200 0α3 0 0.0336

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1 1α2 1α3 1

Viterbi algorithm


A α1 α2 α3

α1 0.8 0.1 0.1α2 0.2 0.7 0.1α3 0.1 0.3 0.6

B β1 β2 β3

α1 0.7 0 0.3α2 0.1 0.9 0α3 0 0.2 0.8

C α1 α2 α3

0.6 0.2 0.2




HMM:

Viterbi algorithm

β1 β3 β1 β3 β3 β2



𝑎 ⋅ 𝐃 𝑗, 𝑛 1


𝑎 ⋅ 𝐃 𝑗, 𝑛 1

𝐃 𝑖, 1 𝑐 ⋅ 𝑏

Initialization

Recursion

D o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2

α1 0.4200 0.1008 0.0564 0.0135 0.0033 0α2 0.0200 0 0.0010 0 0 0.0006α3 0 0.0336 0 0.0045 0.0022 0.0003

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1 1 1 1 1 1α2 1 1 1 1 3α3 1 3 1 3 3

Viterbi algorithm


A α1 α2 α3

α1 0.8 0.1 0.1α2 0.2 0.7 0.1α3 0.1 0.3 0.6

B β1 β2 β3

α1 0.7 0 0.3α2 0.1 0.9 0α3 0 0.2 0.8

C α1 α2 α3

0.6 0.2 0.2




HMM:

D o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2

α1

α2

α3

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1

α2

α3

Viterbi algorithm

𝑖 𝐄 𝑖 , 𝑛

𝑖 argmax∈ :

𝐃 𝑖, 𝑛

Backtracking

β1 β3 β1 β3 β3 β2


D o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2

α1 0.4200 0.1008 0.0564 0.0135 0.0033 0α2 0.0200 0 0.0010 0 0 0.0006α3 0 0.0336 0 0.0045 0.0022 0.0003

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1 1 1 1 1 1α2 1 1 1 1 3α3 1 3 1 3 3

i6 = 2

Viterbi algorithm

D o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2

α1 0.4200 0.1008 0.0564 0.0135 0.0033 0α2 0.0200 0 0.0010 0 0 0.0006α3 0 0.0336 0 0.0045 0.0022 0.0003

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1 1 1 1 1 1α2 1 1 1 1 3α3 1 3 1 3 3

Viterbi algorithm


A α1 α2 α3

α1 0.8 0.1 0.1α2 0.2 0.7 0.1α3 0.1 0.3 0.6

B β1 β2 β3

α1 0.7 0 0.3α2 0.1 0.9 0α3 0 0.2 0.8

C α1 α2 α3

0.6 0.2 0.2




HMM:

D o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2

α1

α2

α3

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1

α2

α3

Viterbi algorithm

𝑖 𝐄 𝑖 , 𝑛

𝑖 argmax∈ :

𝐃 𝑖, 𝑛

Backtracking

β1 β3 β1 β3 β3 β2



A α1 α2 α3

α1 0.8 0.1 0.1α2 0.2 0.7 0.1α3 0.1 0.3 0.6

B β1 β2 β3

α1 0.7 0 0.3α2 0.1 0.9 0α3 0 0.2 0.8

C α1 α2 α3

0.6 0.2 0.2




HMM:

Optimal state sequenceD o1= β1 o2= β3 o3= β1 o4= β3 o5= β3 o6= β2

α1 0.4200 0.1008 0.0564 0.0135 0.0033 0α2 0.0200 0 0.0010 0 0 0.0006α3 0 0.0336 0 0.0045 0.0022 0.0003

E o1= β1 o2= β3 o3= β1 o4= β3 o5= β3

α1 1 1 1 1 1α2 1 1 1 1 3α3 1 3 1 3 3

i6 = 2


β1 β3 β1 β3 β3 β2

S* = (α1,α1,α1,α3,α3,α2)O = (o1,o2,o3,o4,o5,o6)

Input OutputViterbi algorithm

(a) Template Matching (frame-wise)

HMM: Application to Chord Recognition Effect of HMM-based chord estimation and smoothing:

C Dm G C

Time (seconds)Time (seconds)

(b) HMM

HMM: Application to Chord Recognition Parameters: Transition probabilities Estimated from data

State αj

Stat

e α i

Log

prob

abilit

y

Parameters: Transition probabilities Estimated from data

Log

prob

abilit

y

Major chords Minor chords

Maj

or c

hord

sM

inor

cho

rds

HMM: Application to Chord Recognition

Major chords Minor chords

Maj

or c

hord

sM

inor

cho

rds

HMM: Application to Chord Recognition Parameters: Transition probabilities Transposition-invariant

Log

prob

abilit

y

HMM: Application to Chord Recognition Parameters: Transition probabilities Uniform transition matrix (only smoothing)

State αj

Stat

e α i

Log

prob

abilit

y

HMM: Application to Chord Recognition Evaluation on all Beatles songs

0.8

0.75

0.7

0.65

0.6

0.55

0.5

0.45

0.4

0.35

Smoothing length

F-m

easu

re

binary

HMM

1 5 9 13 17 21 25

A

Cm

Em

Am

C

C

E G

E♭

B

E G

Em

C

B

CCmaj7

Chord Recognition: Further Challenges Chord ambiguities

Acoustic ambiguities (overtones) Use advanced templates (model overtones, learned templates) Enhanced chroma (logarithmic compression, overtone reduction)

Tuning inconsistency

Chord Recognition: Public System

Chord recognition Typically: Feature extraction, pattern matching, filtering (HMM) „Out-of-the-box“ solutions (Sonic Visualizer, Chordino plugin)

Sonic Visualizer, Chordino Vamp Plugin(Queen Mary University of London)

Tonal Structures

ChordsCM GM7 Am

Global key



Chord recognition


Segment level

Chord level


MelodyMiddle voices

Bass line

Local key detection

Tonal Structures

ChordsCM GM7 Am

Global key



Chord recognition


Segment level

Chord level


MelodyMiddle voices

Bass line

Local key detection

Local Key Detection

G

D

E

BD♭

E♭

F

A

G♭F♯

A♭

B♭

E ♭mD♯m

B♭mFm

Cm

Gm

Dm

G♯mC♯m

F♯m

Bm

EmAm

C

♭ ♯

Key as an important musical concept (“Symphony in C major”) Modulations → Local approach Key relations: Circle of fifth (keys)

Local Key Detection

Key as an important musical concept (“Symphony in C major”) Modulations → Local approach Diatonic Scales Simplification of keys Perfect-fifth relation

0 diatonic1# diatonic2b diatonic

Circle of fifths (pitches) →

Local Key Detection Example: J.S. Bach, Choral "Durch Dein Gefängnis" (Johannespassion) Score – Piano reduction

Local Key Detection Example: J.S. Bach, Choral "Durch Dein Gefängnis" (Johannespassion) Audio – Waveform (Scholars Baroque Ensemble, Naxos 1994)

Time (seconds)

Tonal Structures: Local Diatonic Scales Example: J.S. Bach, Choral "Durch Dein Gefängnis" (Johannespassion) Audio – Spectrogram (Scholars Baroque Ensemble, Naxos 1994)

Local Key Detection: Chroma Features Example: J.S. Bach, Choral "Durch Dein Gefängnis" (Johannespassion) Audio – Chroma features (Scholars Baroque Ensemble, Naxos 1994)

Local Key Detection: Chroma Smoothing Summarize pitch classes over a certain time

Chroma smoothing Parameters: blocksize b and hopsize h

bb

bh

h

Local Key Detection: Chroma Smoothing Choral (Bach)

Local Key Detection: Chroma Smoothing Choral (Bach) — smoothed with b = 42 seconds and h = 15 seconds

Local Key Detection: Diatonic Scales Choral (Bach) — Re-ordering to perfect fifth series

Local Key Detection: Diatonic Scales Choral (Bach) — Re-ordering to perfect fifth series

Local Key Detection: Diatonic Scales Choral (Bach) — Diatonic Scale Estimation (7 fifths)

4#

Local Key Detection: Diatonic Scales Choral (Bach) — Diatonic Scale Estimation (7 fifths)

5#

Local Key Detection: Diatonic Scales Choral (Bach) — Diatonic Scale Estimation: Multiply chroma values*

Local Key Detection: Diatonic Scales Choral (Bach) — Diatonic Scale Estimation: Multiply chroma values

Local Key Detection: Diatonic Scales Choral (Bach) — Diatonic Scale Estimation

Local Key Detection: Diatonic Scales Choral (Bach) — Diatonic Scale Estimation

4 #(E major)

Local Key Detection: Diatonic Scales Choral (Bach) — Diatonic Scale Estimation: Shift to global key

4 #(E major)

Local Key Detection: Diatonic Scales Choral (Bach) — 0 ≙ 4#

Weiss / Habryka, Chroma-Based Scale Matchingfor Audio Tonality Analysis, CIM 2014

Local Key Detection: Examples L. v. Beethoven – Sonata No. 10 op. 14 Nr. 2, 1. Allegro — 0 ≙ 1

(Barenboim, EMI 1998)

Local Key Detection: Examples R. Wagner, Die Meistersinger von Nürnberg, Vorspiel — 0 ≙ 0

(Polish National Radio Symphony Orchestra, J. Wildner, Naxos 1993)

Source: Wikipedia

With Prof. Rainer Kleinertz, Musicology, Uni Saarland

Richard Wagner, Der Ring des Nibelungen– Four operas, up to 15 hours of music– How is harmony organized at the large scale?– Analyses by A. Lorenz 1924– Hypothesis of „Poetico-musical periods“

DFG-funded Project: Computational Analysis of Harmonic Structures

Up to 18 versions 3 versions manually annotated

No. Conductor Recording hh:mm:ss

1 Barenboim 1991–92 14:54:55

2 Boulez 1980–81 13:44:38

3 Böhm 1967–71 13:39:28

4 Furtwängler 1953 15:04:22

5 Haitink 1988–91 14:27:10

6 Janowski 1980–83 14:08:34

7 Karajan 1967–70 14:58:08

8 Keilberth/Furtwängler 1952–54 14:19:56

9 Krauss 1953 14:12:27

10 Levine 1987–89 15:21:52

11 Neuhold 1993–95 14:04:35

12 Sawallisch 1989 14:06:50

13 Solti 1958–65 14:36:58

14 Swarowsky 1968 14:56:34

15 Thielemann 2011 14:31:13

16 Weigle 2010–12 14:48:46

1 2 3 4

Cross-Version Analysis

Idea: Use analysis results based on different interpretations (versions) Tonal characteristics should not depend on interpretation→ Test reliability of the method

Visualize consistency with gray scheme

Cross-Version Analysis

Act 1

Act 2

Act 3

Die Walküre WWV 86 B

Act 1



Act 1, measures 955–1012 Sieglinde´s narration


Act 1, measures 955–1012 Sieglinde´s narration

Act 1

Act 2

Act 3



Act 3, measures 724–789 Wotan´s punishment


Act 3, measures 724–789 Wotan´s punishment


Exploring Tonal-Dramatic Relationships

Histograms of Analysis over timeD

iato

nic

Scal

es

Das RheingoldWWV 86 A

3897 measures

Die WalküreWWV 86 B

5322 measures

SiegfriedWWV 86 C

6682 measures

GötterdämmerungWWV 86 D

6040 measures

Time (measures)


Sword motif – Die Walküre

Diatonic Scales


Sword motif – Siegfried

Diatonic Scales


Valhalla motif – Das Rheingold

Chords


Valhalla motif – Die Walküre

Chords

Tonal Structures: Complexity Global chroma statistics (audio) 1567 – G. da Palestrina, Missa de Beata Virgine, Credo

G#Eb Bb F C G D A E B F# C#

1

0.8

0.6

0.4

0.2

0

Salie

nce

Pitch class

Circle of fifths →

Tonal Structures: Complexity Global chroma statistics (audio) 1725 – J. S. Bach, Orchestral Suite No. 4 BWV 1069, 1. Ouverture (D major)

G# D# A#F C G D A E B F# C#

1

0.8

0.6

0.4

0.2

0

Salie

nce

Pitch class


Tonal Structures: Complexity Global chroma statistics (audio) 1783 – W. A. Mozart, „Linz“ symphony KV 425, 1. Adagio / Allegro (C major)

G#Eb Bb F C G D A E B F# C#

1

0.8

0.6

0.4

0.2

0

Salie

nce

Pitch class


Tonal Structures: Complexity Global chroma statistics (audio) 1883 – J. Brahms, Symphony No. 3, 1. Allegro con brio (F major)

Ab Eb Bb F C G D A E B F# C#

1

0.8

0.6

0.4

0.2

0

Salie

nce

Pitch class


Tonal Structures: Complexity Global chroma statistics (audio) 1940 – A. Webern, Variations for Orchestra op. 30

Ab Eb Bb F C G D A E B F# C#

1

0.8

0.6

0.4

0.2

0

Salie

nce

Pitch class


Tonal Structures: Complexity Realization of complexity measure Γ Entropy / Flatness measures Distribution over Circle of Fifths

Relating to different time scales!

Γ 0

length 𝑟

Γ 1 𝑟

Γ 1 0 Γ 1

Tonal Structures: Complexity

Weiss / Müller, Quantifying and Visualizing Tonal Complexity, CIM 2014

Com

plex

ityΓ


L. van BeethovenSonata Op. 2, No. 3

1st movement


Op. 2, No. 3 Op. 57, No. 1„Appassionata“

Op. 106, No. 1„Hammerklavier“

Harmony Analysis - International Audio Laboratories Erlangen

Documents