Copyright Nov. 2002, George Tzanetakis Digital Music & Music Processing George Tzanetakis PostDoctoral Fellow Computer Science Department Carnegie Mellon.

Copyright Nov. 2002, George

Tzanetakis

Digital Music & Music Processing

George Tzanetakis PostDoctoral Fellow Computer Science Department Carnegie Mellon University

[email protected]://www.cs.cmu.edu/~gtzan

mailto:[email protected]

http://www.cs.cmu.edu/~gtzan


Tzanetakis

Overview

Music Information Retrieval (MIR) and Computer Audition

Motivation

Techniques

Applications

Computer Music and Sound SynthesisExamples, demos


Tzanetakis

MIR Music History

9000 B.C 1000 1700

1877 1960 2002


Tzanetakis

Music

4 million recorded CD tracks

4000 CDs / month

Mp3 bandwidth %

Global

Pervasive

Persistent

Why ?


Tzanetakis

The future of MIR

Library of all recorded music

Tasks: organize, search, retrieve, classify, recommend, browse, listen, annotate

Examples:


Tzanetakis

Audio MIR Pipeline

Signal Processing Machine Learning Human Computer Interaction

HearingRepresentation

UnderstandingAnalysis

ReactingInteraction


Tzanetakis

Traditional Music Representations


Tzanetakis

Time domain waveform

time

pressure

Decompose into building blocks

time

frequency


Tzanetakis

MIDI

Musical Instrument Digital InterfaceHardware interface

File format

Note eventsDuration, discrete pitch, “instrument”

ExtensionsGeneral MIDI

Notation, OMR, continuous pitch


Tzanetakis

Symbolic vs Audio MIR

Audio

PolyphonicTranscription

Symbolic Representation (MIDI)

MIR

Audio

ComputerAudition

MIR

Machine Learning Models


Tzanetakis

Feature extraction


Tzanetakis

Timbral Texture

Timbre = differentiate sounds of same loudness, pitchTimbral Texture = differentiate mixtures of sounds

Global, statistical and fuzzy properties


Tzanetakis

Spectrum

t

t+1

M

M


Tzanetakis

Fourier Transform P=1/f


Tzanetakis

Short Time Fourier Transform

STFT Filterbank interpretation

Filters Oscillators

Amplitude

Frequency

output


Tzanetakis

Short Time Fourier Transform II

t

t+1

M

M


Tzanetakis

Formants

From “Real Time Synthesis for Interactive Applications”P.Cook, A.K Peters Press, used by permission


Tzanetakis

Linear Prediction Coefficients

Impulses @ f0

White Noise

Source Filter Speech

Lossless tubes


Tzanetakis

MPEG Audio Coding (mp3)

AnalysisFilterbank

Psychoacoustic Model

Available bits

32 linearly spacedbands

Encoder: Slower, ComplicatedDecoder: Faster, Simpler

Perceptual Audio Coding


Tzanetakis

Spectral Shape

t

MCentroidRolloffFluxRMS

Moments…


Tzanetakis

Summary of Timbral Texture Features

Time-Frequency analysis

Signal Processing (STFT, DWT)

Source-filter (LPC)

Perceptual (MP3)

Spectral Shape to feature vector


Tzanetakis

Pitch Content

Harmony-melody = pitch concepts

Music theory score = music

Bridge to symbolic MIR

Automatic music transcription

Non-transcriptive arguments


Tzanetakis

Automatic Pitch Detection

P=1/f

Time-domainFrequency-domain

Perceptual

Zerocrossings Autocorrelation analysis = peaks of function correspond to dominant pitches


Tzanetakis

Pitch Histograms

Jazz IrishChroma - folded

Height - unfolded


Tzanetakis

Automatic Music Transcription

Original Transcribed

Mixture signal Noise suppresionPredominant Pitch

Estimation

Remove detected sound

Estimate # of voices


Tzanetakis

Rhythm

Movement in time

Origins in Poetry (iambic, trochaic)

Foot tapping definition

Hierarchical semi-periodic structure at multiple levels of detail

Links to motion, dance

Running vs global


Tzanetakis

Self similarity

DWT Autocorrelation

Peak Picking

Beat Histograms

EnvelopeExtraction


Tzanetakis

Beat Histograms


Tzanetakis

Analysis

Classification

Segmentation

Similarity Retrieval

Clustering

Thumbnailing

Fingerprinting


Tzanetakis

Analysis Overview

Musical PieceTrajectory Point


Tzanetakis

Query-by-example Content-based Retrieval

Ranked list of k nearest neighbors


Tzanetakis

QBE examples

Rock: BeatlesJazz: Bobby Hutserson Funk: Mano negraWorld: Tibetan singerComputer Music: Paul Lansky

Query Match


Tzanetakis

Automatic Musical Genre Classification

Categorical music descriptions created by humans

Fuzzy boundaries

Statistical propertiesTimbral texture, rhythmic structure, harmonic content

Automatic musical genre classificationEvaluate musical content features

Structure audio collections


Tzanetakis

Genregram demo

Dynamic real-timevisualization for classification

of radio signals


Tzanetakis

Audio segmentation

Detect changes of audio texture


Tzanetakis

Multifeature automatic segmenation methodology

Time series of feature vector v(t)

Detect abrupt changes in trajectory


Tzanetakis

Context & Content Aware User Interfaces

Automatic results not perfect

Music listening is personal and subjective

Browsing vs retrieval

“Overview, zoom and filter, details on demand”, Shneiderman mantra

Adapt UI to music content and contextComputer Audition

Visualization


Tzanetakis

Content and Context

Content ~ fileGenre, male voice, saxophone

Content ~ file, collectionSimilarity

Slow-fast

Multiple visualizationsSame content

Different context


Tzanetakis

Timbregrams

Content & Context Similarity + Time StructurePrincipal Component AnalysisMap feature vectors to color


Tzanetakis

Timbrespaces


Tzanetakis

Islands of Music


Tzanetakis

Auditory Scene Analysis

Copyright Nov. 2002, George Tzanetakis Digital Music & Music Processing George Tzanetakis PostDoctoral Fellow Computer Science Department Carnegie Mellon.

Documents

george tzanetakis music

george tzanetakis fourier

george tzanetakis formants

interaction slide

george tzanetakis spectrum

vector slide

f slide

permission slide