Intelligent real-time music accompaniment for constraint-free improvisation

Intelligent real-time music accompaniment forconstraint-free improvisation

Maximos A. Kaliakatsos–Papakostas1, Andreas Floros2, and Michael N.Vrahatis1

1Computational Intelligence Laboratory (CI Lab), Department of Mathematics,University of Patras, GR-26110 Patras, Greece

2Department of Audio and Visual Arts, Ionian University, GR-49100 Corfu, Greece

8 November 2012

Intelligent music accompaniment 1/37

Music

�melody �harmony

And �together...

Intelligent music accompaniment 2/37 Music accompaniment

Improvisation

Improviser produces novel melodies provided a harmonic context(constraint).

�harmony (constraint) �improviser

Infinite possibilities!

�possible improvisation 1

�possible improvisation 2

...

Intelligent music accompaniment 3/37 Music accompaniment

Constraint-free improvisation

The improviser produces melodies creating hers/his melodic context fromscratch.

improviser harmony

Intelligent music accompaniment 4/37 Intelligent, real-time, constraint free improvization

Intelligent accompaniment for constraint-freeimprovisation

How can the computer understand the improviser’s implied harmoniccontext, and generate proper harmonic response?

this is what thepaper discusses

improviser harmony

Intelligent music accompaniment 5/37 Intelligent, real-time, constraint free improvization

Intelligent accompaniment for constraint-freeimprovisation

How can the computer understand the improviser’s implied harmoniccontext, and generate proper harmonic response?

The computer needs tolisten (understand)

improviser harmony

Intelligent music accompaniment 6/37 Intelligent, real-time, constraint free improvization

Intelligent accompaniment for constraint-freeimprovisation

How can the computer understand the improviser’s implied harmoniccontext, and generate proper harmonic response?

and generate (compose)

improviser harmony

Intelligent music accompaniment 7/37 Intelligent, real-time, constraint free improvization

Music characteristics

Characteristics which the computer should be able to understand andreproduce

humanimproviser

rhythmmodule modulemoduletone intensity

tonelistener

rhythm

listener

intensitylistener

tonegenerator

rhythm

generatorintensity

generator

instrumentspecialization computer

performance

Intelligent music accompaniment 8/37 The proposed framework

Music characteristics

Characteristics which the computer should be able to understand andreproduce

humanimproviser

rhythmmodule modulemoduletone intensity

tonelistener

rhythm

listener

intensitylistener

tonegenerator

rhythm

generatorintensity

generator

instrumentspecialization computer

performance

Intelligent music accompaniment 9/37 The proposed framework

Music characteristics

Characteristics which the computer should be able to understand andreproduce

humanimproviser

rhythmmodule modulemoduletone intensity

tonelistener

rhythm

listener

intensitylistener

tonegenerator

rhythm

generatorintensity

generator

instrumentspecialization computer

performance

Intelligent music accompaniment 10/37 The proposed framework

Music characteristics

Characteristics which the computer should be able to understand andreproduce

humanimproviser

rhythmmodule modulemoduletone intensity

tonelistener

rhythm

listener

intensitylistener

tonegenerator

rhythm

generatorintensity

generator

instrumentspecialization computer

performance

Intelligent music accompaniment 11/37 The proposed framework

Music characteristics

Characteristics which the computer should be able to understand andreproduce

humanimproviser

rhythmmodule modulemoduletone intensity

tonelistener

rhythm

listener

intensitylistener

tonegenerator

rhythm

generatorintensity

generator

instrumentspecialization computer

performance

Intelligent music accompaniment 12/37 The proposed framework

Music characteristics

Characteristics which the computer should be able to understand andreproduce

humanimproviser

rhythmmodule modulemoduletone intensity

tonelistener

rhythm

listener

intensitylistener

tonegenerator

rhythm

generatorintensity

generator

instrumentspecialization computer

performance

Intelligent music accompaniment 13/37 The proposed framework

Tone module

Listen to the chords and their chroma complexity.

Train the logistic map1 to create note sequences with similarcharacteristics (with Differential Evolution).

humanimproviser

chordrecognition

PCPestimation

chord tones

listPCP tones

list

SIEestimation

global SIE

value

tone modulelistener

chord tones

list

PCP tones

list

global SIE

valueselect proper

set of tones

finaltone list

instrument

tonal range

list ofnotes

find properr valuewith DE

logisticmap

r value

soundingnote

tone modulegenerator

(a) listener (b) generator

1xn+1 = r xn (1 − xn)

Intelligent music accompaniment 14/37 The proposed framework

Rhythm module

Listen to the rhythm and compute its rhythm features.

Train an FL–system to create rhythms with similar features (withGenetic Algorithm).

humanimproviser

rhythm module

instrumentspecialization

computer

performance

listener

get rhythmfeatures FL-systems

rhythm modulegenerator

Intelligent music accompaniment 15/37 The proposed framework

Intensity module

Listen to the intensities of notes within a sliding time window.

Assign intensities to upcoming notes, with similar statisticalcharacteristics.

Intelligent music accompaniment 16/37 The proposed framework

Experimental setup and inquiries

Human input

Improviser: human guitar player.

The computer listens to the human performance through MIDI.

Computer output

Intelligent musicians: bass and piano player.

Performance assessmentHow well does the computer adapt to the improviser’s playing style?

Compare qualitative characteristics of the music created by the humanand the computer musicians.

How swiftly is the computer adapted?

Intelligent music accompaniment 17/37 Results

Experimental setup and inquiries

Human input

Improviser: human guitar player.

The computer listens to the human performance through MIDI.

Computer output

Intelligent musicians: bass and piano player.

Performance assessmentHow well does the computer adapt to the improviser’s playing style?

Compare qualitative characteristics of the music created by the humanand the computer musicians.

How swiftly is the computer adapted?

Intelligent music accompaniment 18/37 Results

Experimental setup and inquiries

Human input

Improviser: human guitar player.

The computer listens to the human performance through MIDI.

Computer output

Intelligent musicians: bass and piano player.

Performance assessmentHow well does the computer adapt to the improviser’s playing style?

Compare qualitative characteristics of the music created by the humanand the computer musicians.

How swiftly is the computer adapted?

Intelligent music accompaniment 19/37 Results

Experimental setup and inquiries

Human input

Improviser: human guitar player.

The computer listens to the human performance through MIDI.

Computer output

Intelligent musicians: bass and piano player.

Performance assessmentHow well does the computer adapt to the improviser’s playing style?

Compare qualitative characteristics of the music created by the humanand the computer musicians.

How swiftly is the computer adapted?

Intelligent music accompaniment 20/37 Results

Experimental setup and inquiries

Human input

Improviser: human guitar player.

The computer listens to the human performance through MIDI.

Computer output

Intelligent musicians: bass and piano player.

Performance assessmentHow well does the computer adapt to the improviser’s playing style?

Compare qualitative characteristics of the music created by the humanand the computer musicians.

How swiftly is the computer adapted?

Intelligent music accompaniment 21/37 Results

Experimental setup and inquiries

Human input

Improviser: human guitar player.

The computer listens to the human performance through MIDI.

Computer output

Intelligent musicians: bass and piano player.

Performance assessmentHow well does the computer adapt to the improviser’s playing style?

Compare qualitative characteristics of the music created by the humanand the computer musicians.

How swiftly is the computer adapted?

Intelligent music accompaniment 22/37 Results

Example videos

Intelligent music accompaniment 23/37 Results

Performance assessment metrics

Compare the human and computer generated music by comparing musicalattributes at short fixed time intervals.

Measuring musical attributes

Shannon Information Entropy of the Pitch Class Profile (tonal feature):with the acronym SIE

density of notes within a fixed time window (rhythmic feature)

syncopation of rhythm within 4 music measures (rhythmic feature)

symmetry of rhythm within 4 music measures (rhythmic feature)

intensity of notes within 4 music measures (intensity feature)

Intelligent music accompaniment 24/37 Results

Performance assessment metrics

Compare the human and computer generated music by comparing musicalattributes at short fixed time intervals.

Measuring musical attributes

Shannon Information Entropy of the Pitch Class Profile (tonal feature):with the acronym SIE

density of notes within a fixed time window (rhythmic feature)

syncopation of rhythm within 4 music measures (rhythmic feature)

symmetry of rhythm within 4 music measures (rhythmic feature)

intensity of notes within 4 music measures (intensity feature)

Intelligent music accompaniment 25/37 Results

Performance assessment metrics

Compare the human and computer generated music by comparing musicalattributes at short fixed time intervals.

Measuring musical attributes

Shannon Information Entropy of the Pitch Class Profile (tonal feature):with the acronym SIE

density of notes within a fixed time window (rhythmic feature)

syncopation of rhythm within 4 music measures (rhythmic feature)

symmetry of rhythm within 4 music measures (rhythmic feature)

intensity of notes within 4 music measures (intensity feature)

Intelligent music accompaniment 26/37 Results

Performance assessment metrics

Compare the human and computer generated music by comparing musicalattributes at short fixed time intervals.

Measuring musical attributes

Shannon Information Entropy of the Pitch Class Profile (tonal feature):with the acronym SIE

density of notes within a fixed time window (rhythmic feature)

syncopation of rhythm within 4 music measures (rhythmic feature)

symmetry of rhythm within 4 music measures (rhythmic feature)

intensity of notes within 4 music measures (intensity feature)

Intelligent music accompaniment 27/37 Results

Performance assessment metrics

Compare the human and computer generated music by comparing musicalattributes at short fixed time intervals.

Measuring musical attributes

Shannon Information Entropy of the Pitch Class Profile (tonal feature):with the acronym SIE

density of notes within a fixed time window (rhythmic feature)

syncopation of rhythm within 4 music measures (rhythmic feature)

symmetry of rhythm within 4 music measures (rhythmic feature)

intensity of notes within 4 music measures (intensity feature)

Intelligent music accompaniment 28/37 Results

Performance assessment metrics

Compare the human and computer generated music by comparing musicalattributes at short fixed time intervals.

Measuring musical attributes

Shannon Information Entropy of the Pitch Class Profile (tonal feature):with the acronym SIE

density of notes within a fixed time window (rhythmic feature)

syncopation of rhythm within 4 music measures (rhythmic feature)

symmetry of rhythm within 4 music measures (rhythmic feature)

intensity of notes within 4 music measures (intensity feature)

Intelligent music accompaniment 29/37 Results

Performance assessment metrics

Compare the human and computer generated music by comparing musicalattributes at short fixed time intervals.

Measuring musical attributes

Shannon Information Entropy of the Pitch Class Profile (tonal feature):with the acronym SIE

density of notes within a fixed time window (rhythmic feature)

syncopation of rhythm within 4 music measures (rhythmic feature)

symmetry of rhythm within 4 music measures (rhythmic feature)

intensity of notes within 4 music measures (intensity feature)

Intelligent music accompaniment 30/37 Results

Performance assessment metrics

SIE

guitarpianobass

Rhythm density

guitarpianobass

Rhythm syncopation

guitarpianobass

Rhythm symmetry

guitarpianobass

Intensities

guitarpianobass

Intelligent music accompaniment 31/37 Results

Performance assessment in improvisation 1

improvisation 1no delay delay

piano bass piano bassSIE 0.4280 0.5571 0.4240 0.5735

SIE (MA) 0.6516 0.7700 0.6829 0.8320density 0.4659 0.5321 0.7045 0.8557

density (MA) 0.5416 0.6053 0.7771 0.9064syncopation 0.1789 0.4802 0.2230 0.3418

syncopation (MA) 0.4542 0.6417 0.6551 0.7188symmetry -0.2060 -0.3752 -0.0050 0.4130

symmetry (MA) 0.1425 0.0314 0.4222 0.2274intensity 0.6696 0.6731 — —

Intelligent music accompaniment 32/37 Results

Performance assessment in improvisation 2

improvisation 2no delay delay

piano bass piano bassSIE 0.1949 0.1236 0.2601 0.0551

SIE (MA) 0.6189 0.4954 0.7198 0.5529density 0.5229 0.6316 0.7907 0.7220

density (MA) 0.7090 0.7133 0.8941 0.8520syncopation 0.0326 0.2958 0.0977 0.2191

syncopation (MA) 0.5455 0.5357 0.7203 0.6710symmetry -0.1455 0.0474 0.2701 0.0616

symmetry (MA) 0.2125 0.3113 0.4305 0.4902intensity 0.5459 0.5745 — —

Intelligent music accompaniment 33/37 Results

Performance assessment in improvisation 3

improvisation 3no delay delay

piano bass piano bassSIE 0.3686 0.2823 0.4133 0.2826

SIE (MA) 0.6455 0.4098 0.7308 0.4166density 0.6011 0.5285 0.8002 0.7011

density (MA) 0.7139 0.5964 0.8951 0.7687syncopation 0.0152 0.0764 -0.0298 0.1130

syncopation (MA) 0.4374 0.2382 0.6900 0.3164symmetry -0.1904 0.0927 0.1161 0.0235

symmetry (MA) 0.1348 0.2161 0.4947 0.2656intensity 0.6242 0.6345 — —

Intelligent music accompaniment 34/37 Results

Overall tonal impression

Comparing the pitch class profile of the entire recordings by each “musician”.

improvisation 1 improvisation 2 improvisation 3guitar piano bass guitar piano bass guitar piano bass

guitar 1.0000 0.9208 0.9160 1.0000 0.9583 0.9806 1.0000 0.8418 0.8490piano 0.9208 1.0000 0.9722 0.9583 1.0000 0.9754 0.8418 1.0000 0.9213bass 0.9160 0.9722 1.0000 0.9806 0.9754 1.0000 0.8490 0.9213 1.0000

Intelligent music accompaniment 35/37 Results

Concluding discussion

Summary

The presented system:

provides intelligent automatic accompaniment to a human improviserwithout any prior musical considerations

adapts to the human improviser’s tonal, rhythmic and intensityperforming style and composes novel music

Future improvements

examine whether the training process could be based on the MovingAverage time series of features

the listener submodules could enhanced by the introduction of newfeatures that incorporate further descriptive knowledge

the generator submodules could be further enhanced by utilizing moresophisticated intelligent and adaptive techniques

Intelligent music accompaniment 36/37 Conclusions

Concluding discussion

Summary

The presented system:

provides intelligent automatic accompaniment to a human improviserwithout any prior musical considerations

adapts to the human improviser’s tonal, rhythmic and intensityperforming style and composes novel music

Future improvements

examine whether the training process could be based on the MovingAverage time series of features

the listener submodules could enhanced by the introduction of newfeatures that incorporate further descriptive knowledge

the generator submodules could be further enhanced by utilizing moresophisticated intelligent and adaptive techniques

Intelligent music accompaniment 37/37 Conclusions

Concluding discussion

Summary

The presented system:

provides intelligent automatic accompaniment to a human improviserwithout any prior musical considerations

adapts to the human improviser’s tonal, rhythmic and intensityperforming style and composes novel music

Future improvements

examine whether the training process could be based on the MovingAverage time series of features

the listener submodules could enhanced by the introduction of newfeatures that incorporate further descriptive knowledge

the generator submodules could be further enhanced by utilizing moresophisticated intelligent and adaptive techniques

Intelligent music accompaniment 38/37 Conclusions

Concluding discussion

Summary

The presented system:

provides intelligent automatic accompaniment to a human improviserwithout any prior musical considerations

adapts to the human improviser’s tonal, rhythmic and intensityperforming style and composes novel music

Future improvements

examine whether the training process could be based on the MovingAverage time series of features

the listener submodules could enhanced by the introduction of newfeatures that incorporate further descriptive knowledge

the generator submodules could be further enhanced by utilizing moresophisticated intelligent and adaptive techniques

Intelligent music accompaniment 39/37 Conclusions

Concluding discussion

Summary

The presented system:

provides intelligent automatic accompaniment to a human improviserwithout any prior musical considerations

adapts to the human improviser’s tonal, rhythmic and intensityperforming style and composes novel music

Future improvements

examine whether the training process could be based on the MovingAverage time series of features

the listener submodules could enhanced by the introduction of newfeatures that incorporate further descriptive knowledge

the generator submodules could be further enhanced by utilizing moresophisticated intelligent and adaptive techniques

Intelligent music accompaniment 40/37 Conclusions

Concluding discussion

Summary

The presented system:

provides intelligent automatic accompaniment to a human improviserwithout any prior musical considerations

adapts to the human improviser’s tonal, rhythmic and intensityperforming style and composes novel music

Future improvements

examine whether the training process could be based on the MovingAverage time series of features

the listener submodules could enhanced by the introduction of newfeatures that incorporate further descriptive knowledge

the generator submodules could be further enhanced by utilizing moresophisticated intelligent and adaptive techniques

Intelligent music accompaniment 41/37 Conclusions

Concluding discussion

Summary

The presented system:

provides intelligent automatic accompaniment to a human improviserwithout any prior musical considerations

adapts to the human improviser’s tonal, rhythmic and intensityperforming style and composes novel music

Future improvements

examine whether the training process could be based on the MovingAverage time series of features

the listener submodules could enhanced by the introduction of newfeatures that incorporate further descriptive knowledge

the generator submodules could be further enhanced by utilizing moresophisticated intelligent and adaptive techniques

Intelligent music accompaniment 42/37 Conclusions

Thank You!Question-Comments

[email protected]://sites.google.com/site/maximoskphttp://cilab.math.upatras.gr

Intelligent music accompaniment 43/37 Conclusions