Top Banner
Julius Smith AES-2006 Heyser Lecture – 1 / 84 History and Practice of Digital Sound Synthesis Julius Smith CCRMA, Stanford University AES-2006 Heyser Lecture October 6, 2006
85

History and Practice of Digital Sound Synthesis

Dec 30, 2016

Download

Documents

trinhthien
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: History and Practice of Digital Sound Synthesis

Julius Smith AES-2006 Heyser Lecture – 1 / 84

History and Practice of Digital Sound Synthesis

Julius SmithCCRMA, Stanford University

AES-2006 Heyser Lecture

October 6, 2006

Page 2: History and Practice of Digital Sound Synthesis

Overview

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 2 / 84

Page 3: History and Practice of Digital Sound Synthesis

Outline

Overview

• Outline

• CCRMA Perspective

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 3 / 84

Digital sound synthesis approaches in approximate historical order:

• Wavetable (one period)• Subtractive• Additive• Frequency Modulation (FM)• Sampling• Spectral Modeling• Physical Modeling

Some connections with audio coding will be noted

Emphasis:

• Sound examples• Block diagrams• Historical notes

Page 4: History and Practice of Digital Sound Synthesis

CCRMA Perspective

Overview

• Outline

• CCRMA Perspective

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 4 / 84

The Knoll, Stanford University

Page 5: History and Practice of Digital Sound Synthesis

Early Digital Sound Synthesis

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 5 / 84

Page 6: History and Practice of Digital Sound Synthesis

Wavetable Synthesis in Music I-V (1957-1969)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 6 / 84

Page 7: History and Practice of Digital Sound Synthesis

Music V Scripting Language (“Note Cards”)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 7 / 84

• Essentially Supported in MPEG-4 Structured Audio OrchestraLanguage (SAOL) (Music V → csound→ SAOL)

• “Encoding sounds” as “instruments” is hard, in general

Page 8: History and Practice of Digital Sound Synthesis

Kelly-Lochbaum Vocal Tract Model

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 8 / 84

k1 − k1

1 − k1

1 + k1

R1

z− 21

z− 21

z− 21

z− 21

…kM − kM

1 − kM

1 + kM

RM

Glottal PulseTrain or Noise

e(n)SpeechOutput

y(n)

(UnusedAllpassOutput)

Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical)

…e(n) y(n)

John L. Kelly and Carol Lochbaum (1962)

Page 9: History and Practice of Digital Sound Synthesis

Sound Example

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 9 / 84

“Bicycle Built for Two”: (WAV) (MP3)

• Vocal part by Kelly and Lochbaum (1961)• Musical accompaniment by Max Mathews• Computed on an IBM 704• Based on Russian speech-vowel data from Gunnar Fant’s book• Probably the first digital physical-modeling synthesis sound

example by any method• Inspired Arthur C. Clarke to adapt it for “2001: A Space Odyssey”

— the computer’s “first song”

Page 10: History and Practice of Digital Sound Synthesis

Classic Additive-Synthesis Analysis (Heterodyne Comb)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 10 / 84

John Grey 1975 — CCRMA Tech. Reports 1 & 2(CCRMA “STANM” reports — available online)

Page 11: History and Practice of Digital Sound Synthesis

Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 11 / 84

John Grey 1975 — CCRMA Tech. Reports 1 & 2(CCRMA “STANM” reports — available online)

Page 12: History and Practice of Digital Sound Synthesis

Classic Additive Synthesis Diagram

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 12 / 84

noise

FIR

A1(t) A2(t) A3(t) A4(t)f1(t) f2(t) f3(t) f4(t)

Σ

y(t) =4

i=1

Ai(t) sin

[∫ t

0

ωi(t)dt + φi(0)

]

Page 13: History and Practice of Digital Sound Synthesis

Classic Additive-Synthesis Examples

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 13 / 84

• Bb Clarinet• Eb Clarinet• Oboe• Bassoon• Tenor Saxophone• Trumpet• English Horn• French Horn• Flute

• All of the above• Independently synthesized set

(Synthesized from original John Grey data)

Page 14: History and Practice of Digital Sound Synthesis

Frequency Modulation (FM) Synthesis

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 14 / 84

FM synthesis is normally used as a spectral modeling technique.

• Discovered and developed (1970s) by John M. Chowning(CCRMA Founding Director)

• Key paper: JAES 1973 (vol. 21, no. 7)• Commercialized by Yamaha Corporation:

DX-7 synthesizer (1983) OPL chipset (SoundBlaster PC sound card) Cell phone ring tones

• On the physical modeling front, synthesis of vibrating-stringwaveforms using finite differences started around this time:Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)

Page 15: History and Practice of Digital Sound Synthesis

FM Formula

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 15 / 84

x(t) = Ac sin[ωct + φc + Am sin(ωmt + φm)]

where

(Ac, ωc, φc) specify the carrier sinusoid(Am, ωm, φm) specify the modulator sinusoid

Can also be called phase modulation

Page 16: History and Practice of Digital Sound Synthesis

Simple FM “Brass” Patch (1970–)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 16 / 84

Jean-Claude Risset observation (1964–1969):Brass bandwidth ∝ amplitude

A F

FA

Out

fc = f0

fm = f0

g

Page 17: History and Practice of Digital Sound Synthesis

FM Harmonic Amplitudes (Bessel Function of First Kind)

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 17 / 84

Harmonic number k, FM index β:

0

2

4

6

8

10

05

1015

2025

30

−0.5

0

0.5

1

Order k

Argument β

J k(β)

Page 18: History and Practice of Digital Sound Synthesis

Frequency Modulation (FM) Examples

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 18 / 84

All examples by John Chowning unless otherwise noted:

• FM brass synthesis

Low Brass example Dexter Morril’s FM Trumpet

• FM singing voice (1978)Each formant synthesized using an FM operator pair(two sinusoidal oscillators)

Chorus Voices Basso Profundo

• Other early FM synthesis

Clicks and Drums Big Bell String Canon

Page 19: History and Practice of Digital Sound Synthesis

FM Voice

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 19 / 84

FM voice synthesis can be viewed as compressed modeling ofspectral formants

Frequency f0

Mag

nitu

de

Carrier 1 Carrier 2

Carrier 3

Modulation Frequency (all three)

Page 20: History and Practice of Digital Sound Synthesis

Sampling Synthesis History

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 20 / 84

• 1979 - Fairlight Computer Music Instrument - 8-bit

First commercial sampler Eight voices, 8 bits, 64 KB (4 sec) RAM, 16 kHz (mono) Editing, looping, mixing One could draw waveforms and additive-synthesis amplitude

envelopes (for each harmonic) with a light pen $25,000–$36,000!

• 1981 - E-mu Systems Emulator

First “affordable” sampler ($10,000) Eight voices, 128 K RAM, 8-bit, 80 lb.

• 1986 - Ensoniq Mirage

Breakthrough price-point ($1695) Eight voices, 144 K RAM, 8-bit

Page 21: History and Practice of Digital Sound Synthesis

Modern Sampled Piano

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 21 / 84

Example: 1

• 40 Gigabytes on ten DVDs (three sampled pianos)• Every key sampled• 4–10 “velocity layers”• Separate recordings with soft pedal down• Separate “release” recordings, for multiple striking velocities

1Synthogy Ivory, $349 (Electronic Musician, October 2006)

Page 22: History and Practice of Digital Sound Synthesis

Fundamental Problem with Sampling Synthesis

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 22 / 84

Piano timbre is determined by

• key number (1 byte)• key velocity (2 bytes more than enough)• pedal state (1 bit [or byte] per pedal)

Piano control is relatively low-dimensional:

• Less than six bytes of information per note played• No continuous controls (typically)• Ratio of total sampled data to one note of control data

≈ one billion

Page 23: History and Practice of Digital Sound Synthesis

Now consider bowed strings

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 23 / 84

Control parameters:

• Left-hand finger position(s)• Left-hand vibrato• Bow velocity• Bow force• Bow position• Bow angle• Shoulder damping• Instrument orientation• Player motion (within a room)

Page 24: History and Practice of Digital Sound Synthesis

Difficulty of sampling bowed strings

Overview

Early Digital Synthesis

• Music V

• KL Music

• “Daisy”

• Additive Analysis

• Additive Synthesis

• FM Synthesis

• FM Formula

• FM Patch

• FM Spectra

• FM Examples

• FM Voice

• Sampling Synthesis

• Modern Example

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 24 / 84

• Bowed-string control is infinite-dimensional in principle• Many time-varying functions — “gestures”

(we counted more than 10)• Complete sampling of bowed strings on the level of pianos has

apparently never been done• Rule-driven navigation of the most useful recorded playing

regimes has worked well (e.g., Synful Orchestra)• Model-based approaches greatly reduce data requirements:

Spectral models (inspired by sound perception) Physical models (model the sound source)

Page 25: History and Practice of Digital Sound Synthesis

Spectral Modeling Synthesis(Historical Summary)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 25 / 84

Page 26: History and Practice of Digital Sound Synthesis

Classic Vocoder Analysis & Resynthesis (Dudley 1939)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 26 / 84

f

A

f

A

f

A

Analysis Synthesis

Data Compression,Transmission,Storage,Manipulation,Noise reduction, ...

Processing

magnitude, ormagnitude andphase extraction

x(t)

x0(t)

x1(t)

xN−1

x(t)

x0(t)

x1(t)

xN−1

Page 27: History and Practice of Digital Sound Synthesis

Phase Vocoder Channel Model

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 27 / 84

ωk ω0

Channel FilterResponse

ak

∆ωk

Analysis Model Synthesis Model

ak(t) ωk+∆ωk(t)

A F

Out

Sine Osc

• Early “channel vocoder” implementations (hardware) onlymeasured amplitude ak(t) (Dudley 1939)

• The “phase vocoder” (Flanagan and Golden 1966) added phasetracking in each channel

• Portnoff (1976) developed the FFT phase vocoder,which replaced the heterodyne comb in computer-musicadditive-synthesis analysis (James A. Moorer)

• Inverse FFT synthesis (Rodet and Depalle 1992) gave fastersinusoidal oscillator banks

Page 28: History and Practice of Digital Sound Synthesis

Amplitude and Frequency Envelopes

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 28 / 84

ak(t)

∆ωk(t) = φk(t)

0

t

t

Page 29: History and Practice of Digital Sound Synthesis

Channel Vocoder Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 29 / 84

• Original

• 10 channels, sine carriers• 10 channels, narrowband-noise carriers

• 26 channels, sine carriers• 26 channels, narrowband-noise carriers• 26 channels, narrowband-noise carriers, channels reversed

• Phase Vocoder: Identity system in absence of modifications

• The FFT Phase Vocoder next transitioned to the Short-TimeFourier Transform (STFT) (Allen and Rabiner 1977)

Page 30: History and Practice of Digital Sound Synthesis

Tracking Spectral Peaks in the Short-Time Fourier Transform

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 30 / 84

dB magPeaktracking

QuadraticPeakInterpolation

Frequencies

Amplitudes

Phases

window w(n)

s(t)

atan

FFT

• STFT peak tracking at CCRMA: mid-1980s (PARSHL program)• Motivated by vocoder analysis of piano tones• Influences: STFT (Allen and Rabiner 1977),

ADEC (1977), MAPLE (1979)• Independently developed for speech coding by McAulay and

Quatieri at Lincoln Labs

Page 31: History and Practice of Digital Sound Synthesis

Example Spectral Trajectories

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 31 / 84

f

t

Page 32: History and Practice of Digital Sound Synthesis

Sines + Noise Synthesis (1989)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 32 / 84

PSfragA1(t) A2(t) A3(t) A4(t)f1(t) f2(t) f3(t) f4(t)

∑white noiseu(t) filter(t)

ht(τ)

y(t) =4

i=1

Ai(t) cos[

∫ t

0ωi(t)dt + φi(0)

]

+ (ht ∗ u)(t)

Page 33: History and Practice of Digital Sound Synthesis

Sines + Noise Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 33 / 84

Xavier Serra 1989 thesis demos (Sines + Noise signal modeling)

• Piano

Original Sinusoids alone Residual after sinusoids removed Sines + noise model

• Voice

Original Sinusoids Residual Synthesis

Page 34: History and Practice of Digital Sound Synthesis

Musical Effects with Sines+Noise Models (Serra 1989)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 34 / 84

• Piano Effects

Pitch downshift one octave Pitch flattened Varying partial stretching

• Voice Effects

Frequency-scale by 0.6 Frequency-scale by 0.4 and stretch partials Variable time-scaling, deterministic to stochastic

Page 35: History and Practice of Digital Sound Synthesis

Cross-Synthesis with Sines+Noise Models (Serra 1989)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 35 / 84

• Voice “modulator”

• Creaking ship’s mast “carrier”• Voice-modulated creaking mast• Same with modified spectral envelopes

Page 36: History and Practice of Digital Sound Synthesis

Sines + Transients Sound Examples (Serra 1989)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 36 / 84

In this technique, the sinusoidal sum is phase-matched at thecross-over point only (with no cross-fade).

• Marimba

Original Sinusoidal model Original attack, followed by sinusoidal model

• Piano

Original Sinusoidal model Original attack, followed by sinusoidal model

Page 37: History and Practice of Digital Sound Synthesis

Multiresolution Sines + Noise + Transients (Levine 1998)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 37 / 84

Why Model Transients Separately?

• Sinusoids efficiently model spectral peaks over time• Filtered noise efficiently models spectral residual vs. t• Neither is good for abrupt transients in the waveform• Phase-matched oscillators are expensive• More efficient to switch to a transient model during transients• Need sinusoidal phase matching at the switching times

Transient models:

• Original waveform slice (1988)• Wavelet expansion (Ali 1996)• MPEG-2 AAC (with short window) (Levine 1998)• Frequency-domain LPC

(time-domain amplitude envelope) (Verma 2000)

Page 38: History and Practice of Digital Sound Synthesis

Time Scale Modification of Sines + Noise + Transients Models

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 38 / 84

transien

ts

transien

ts

transien

ts

transien

ts

sines +noise

sines +noisesines +noise

sines +noise

sines +noise

sines +noise

sines +noise

sines +noise

sines +noise

original signal

time-scaledsignal

time

Time-Scale Modification (TSM) becomes well defined :

• Transients are translated in time• Sinusoidal envelopes are scaled in time• Noise-filter envelopes also scaled in time• Dual of TSM is frequency scaling

Page 39: History and Practice of Digital Sound Synthesis

Sines + Noise + Transients Time-Frequency Map

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 39 / 84

0 50 100 150 200 250−1

−0.5

0

0.5

1

time [milliseconds]

ampl

itude

0 50 100 150 200 2500

2

4

6

8

10

12

14

16

freq

uenc

y [k

Hz]

(Levine 1998)

Page 40: History and Practice of Digital Sound Synthesis

Corresponding Analysis Windows

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 40 / 84

0 50 100 150 200 250time [milliseconds]

ampl

itude

tran

sien

thi

gh o

ctav

em

iddl

e oc

tave

low

oct

ave

Page 41: History and Practice of Digital Sound Synthesis

Quasi-Constant-Q (Wavelet) Time-Frequency Map

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 41 / 84

0 50 100 150 200 250

−2

−1

0

1

2

x 104

time [milliseconds]

ampl

itude

0 50 100 150 200 2500

2

4

6

8

10

12

14

16

freq

uenc

y [k

Hz]

Page 42: History and Practice of Digital Sound Synthesis

Bark-Band Noise Modeling at High Frequencies (Levine 1998)

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 42 / 84

0 500 1000 1500 2000 2500 3000 3500 4000 450030

35

40

45

50

55

60

65

70

75

80

85

frequency [Hz]

mag

nitu

de [d

B]

Page 43: History and Practice of Digital Sound Synthesis

Amplitude Envelope for One Noise Band

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 43 / 84

0 50 100 150 200 250 300 35040

50

60

70

80

Orig

inal

Mag

. [dB

]

0 50 100 150 200 250 300 35045

50

55

60

65

70

75

80

time [milliseconds]

LSA

Mag

. [dB

]

For more information, see Scott Levine’s thesis.22http://ccrma.stanford.edu/~scottl/thesis.html

Page 44: History and Practice of Digital Sound Synthesis

Sines + Noise + Transients Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 44 / 84

Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)(http://ccrma.stanford.edu/~scottl/thesis.html)

Mozart’s Le Nozze di Figaro

• Original• Compressed using MPEG-AAC at 32 kbps• Compressed using sines+transients+noise at 32 kbps

• Multiresolution sinusoids alone• Residual Bark-band noise• Transform-coded transients (AAC)• Bark-band noise above 5 kHz

Page 45: History and Practice of Digital Sound Synthesis

Rock Example

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 45 / 84

Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)(http://ccrma.stanford.edu/~scottl/thesis.html)

“It Takes Two” by Rob Base & DJ E-Z Rock

• Original• MPEG-AAC at 32 kbps• Sines+transients+noise at 32 kbps

• Multiresolution sinusoids• Residual Bark-band noise• Transform-coded transients (AAC)• Bark-band noise above 5 kHz

Page 46: History and Practice of Digital Sound Synthesis

Time Scale Modification using Sines + Noise + Transients

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 46 / 84

Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)(http://ccrma.stanford.edu/~scottl/thesis.html)

Time-Scale Modification (pitch unchanged)

• S+N+T time-scale factors [2.0, 1.6, 1.2, 1.0, 0.8, 0.6, 0.5]

S+N+T Pitch Shifting (timing unchanged)

• Pitch-scale factors [0.89, 0.94, 1.00, 1.06, 1.12]

Page 47: History and Practice of Digital Sound Synthesis

Spectral Modeling History Highlights

Julius Smith AES-2006 Heyser Lecture – 47 / 84

• Fourier’s theory (1822)• Teleharmonium (1906)• Hammond organ (1930s)• Channel Vocoder (1939)• Phase Vocoder (1966)• “Additive Synthesis” (1969)• FFT Phase Vocoder (1976)• Sinusoidal Modeling

(1977,1979,1985)• Sines+Noise (1989)• Sines+Transients (1989)• Sines+Noise+Transients (1998)

Perceptual audio coding:

• Princen-Bradley filterbank (1986)• K. Brandenburg thesis (1989)• Auditory masking usage• Dolby AC2• Musicam• ASPEC• MPEG-I,II,IV

(incl. S+N+T “parametric sounds”)

Page 48: History and Practice of Digital Sound Synthesis

Future Prospects

Overview

Early Digital Synthesis

Spectral Modeling

• Vocoder

• Vocoder Examples

• Sinusoidal Modeling

• Spectral Trajectories

• Sines + Noise

• S+N Examples

• S+N FX

• S+N XSynth

• Sines + Transients

• S + N + Transients

• S+N+T TSM

• S+N+T Freq Map

• S+N+T Windows

• HF Noise Modeling

• HF Noise Band

• S+N+T Examples

• SM Summary

• Spec Future

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 48 / 84

Observations:

• Sinusoidal modeling of sound is “Unreasonably Effective”• Basic “auditory masking” discards ≈ 90% information• Interesting neuroscience observation:

“... most neurons in the primary auditory cortex A1are silent most of the time ...”

(from “Sparse Time-Frequency Representations”, Gardner andMagnesco, PNAS:103(16), April 2006)

• What is a true and correct “psychospectral model” for sound?

The cochlea of the ear is a real-time spectrum analyzer How is the “ear’s spectrogram” represented at higher levels?

Page 49: History and Practice of Digital Sound Synthesis

Physical Modeling Synthesis(Historical Summary)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 49 / 84

Page 50: History and Practice of Digital Sound Synthesis

Kelly-Lochbaum Vocal Tract Model

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 50 / 84

k1 − k1

1 − k1

1 + k1

R1

z− 21

z− 21

z− 21

z− 21

…kM − kM

1 − kM

1 + kM

RM

Glottal PulseTrain or Noise

e(n)SpeechOutput

y(n)

(UnusedAllpassOutput)

Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical)

…e(n) y(n)

John L. Kelly and Carol Lochbaum (1962)

Page 51: History and Practice of Digital Sound Synthesis

Digital Waveguide Models (1985)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 51 / 84

Lossless digital waveguide∆= bidirectional delay line

at some wave impedance R

z−N

z−N

R

Useful for efficient models of

• strings• bores• plane waves• conical waves

Page 52: History and Practice of Digital Sound Synthesis

Signal Scattering

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 52 / 84

Signal scattering is caused by a change in wave impedance R:

z−N

z−N

k1 − k1

1 − k1

1 + k1

z−N

z−N

R1 R2

RRRR

k12

121 +

−=

If the wave impedance changes every spatial sample, theKelly-Lochbaum vocal-tract model results.

Page 53: History and Practice of Digital Sound Synthesis

Ideal Plucked String (Displacement Waves)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 53 / 84

(x = 0) (x = L)

y (n+N/2)

-1“Bridge”

y (n)+

“Nut”

-y (n)-

-1

y (n-N/2)+

(x = Pluck Position)

• Load each delay line with half of initial string displacement• Sum of upper and lower delay lines = string displacement

Page 54: History and Practice of Digital Sound Synthesis

Ideal Struck String (Velocity Waves)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 54 / 84

(x = 0) (x = L)

v (n+N/2)

-1“Bridge”

v (n)+

“Nut”

-v (n)-

-1

v (n-N/2)+

(x = Hammer Position)

c

c

Hammer strike = momentum transfer = velocity step:

mhvh(0−) = (mh + ms)vs(0+)

Page 55: History and Practice of Digital Sound Synthesis

Karplus-Strong (KS) Algorithm (1983)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 55 / 84

N samples delayOutput y (n)+

z 1-

1/2

1/2

y (n-N)+

• Discovered (1978) as “self-modifying wavetable synthesis”• Wavetable is preferably initialized with random numbers

Page 56: History and Practice of Digital Sound Synthesis

Karplus-Strong Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 56 / 84

• “Vintage” 8-bit sound examples:

• Original Plucked String: (WAV) (MP3)• Drum: (WAV) (MP3)• Stretched Drum: (WAV) (MP3)

Page 57: History and Practice of Digital Sound Synthesis

EKS Algorithm (Jaffe-Smith 1983)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 57 / 84

Hβ(z)

Hρ(z) Hs(z)

HL(z)Hp(z)

Hd(z)

z−N

N = pitch period (2× string length) in samples

Hp(z) =1 − p

1 − p z−1= pick-direction lowpass filter

Hβ(z) = 1 − z−βN = pick-position comb filter, β ∈ (0, 1)

Hd(z) = string-damping filter (one/two poles/zeros typical)

Hs(z) = string-stiffness allpass filter (several poles and zeros)

Hρ(z) =ρ(N) − z−1

1 − ρ(N) z−1= first-order string-tuning allpass filter

HL(z) =1 − RL

1 − RL z−1= dynamic-level lowpass filter

Page 58: History and Practice of Digital Sound Synthesis

STK EKS Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 58 / 84

• Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, andothers — distributed by CCRMA:Google search: STK ToolKit

STK Plucked String: (WAV) (MP3)

• Plucked String 1: (WAV) (MP3)• Plucked String 2: (WAV) (MP3)• Plucked String 3: (WAV) (MP3)

Page 59: History and Practice of Digital Sound Synthesis

EKS Sound Example (1988)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 59 / 84

Bach A-Minor Concerto—Orchestra Part: (WAV) (MP3)

• Executed in real time on one Motorola DSP56001(20 MHz clock, 128K SRAM)

• Developed for the NeXT Computer introduction at DaviesSymphony Hall, San Francisco, 1988

• Solo violin part was played live by Dan Kobialka of the SanFrancisco Symphony

Page 60: History and Practice of Digital Sound Synthesis

Digital Waveguide Single Reed, Cylindrical Bore Model (1986)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 60 / 84

Bell

MouthPressure

EmbouchureOffset

Reed to Bell Delay( )npm

2

BoreReed

ReflectionFilter

OutputFilter

Bell to Reed Delay

( )np+b

( )np−b

-

-h∆

+

hm

-

Reed Table

Digital waveguide clarinet

• Control variable = mouth half-pressure• Total reed cost = two subtractions, one multiply, and one table

lookup per sample

Page 61: History and Practice of Digital Sound Synthesis

Digital Waveguide Wind Instrument Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 61 / 84

• STK Clarinet: (WAV) (MP3)Google search: STK clarinet

Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, andothers — distributed by CCRMA:Google search: STK ToolKit

• Staccato Systems Slide Flute(based on STK flute, ca. 1995): (WAV) (MP3)

• Yamaha VL1 “Virtual Lead” synthesizer demos (1994):

• Shakuhachi: (WAV) (MP3)• Oboe and Bassoon: (WAV) (MP3)• Tenor Saxophone: (WAV) (MP3)

Page 62: History and Practice of Digital Sound Synthesis

Digital Waveguide Bowed Strings (1986)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 62 / 84

Bridge-Body

Bow Force

Bow to Bridge Delay

String

ReflectionFilter

BodyFilter

Bridge to Bow Delay

Nut to Bow Delay

Bow to Nut Delay

-1

String BowNut Air

Bow Velocity

-

v+ls,

v+rs,

v∆+ ρ

v−ls,

v−rs,

- *vb

Bow Table

• Reflection filter summarizes all losses per period(due to bridge, bow, finger, etc.)

• Bow-string junction = memoryless lookup table(or segmented polynomial)

Page 63: History and Practice of Digital Sound Synthesis

“Electric Cello” Sound Examples (Peder Larson)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 63 / 84

• Staccato Notes: (WAV) (MP3)(short strokes of high bow pressure, as from a bouncing bow)

• Bach’s First Suite for Unaccompanied Cello: (WAV) (MP3)

Page 64: History and Practice of Digital Sound Synthesis

Soft Clipper

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 64 / 84

f(x) =

−2

3, x ≤ −1

x −x3

3, −1 ≤ x ≤ 1

2

3, x ≥ 1

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8x=−1:0.01:1; plot([−(2/3)*ones(1,100), x−x.3/3, (2/3)*ones(1,100)])

x(n)

f(x(n

))

Page 65: History and Practice of Digital Sound Synthesis

Amplifier Distortion + Amplifier Feedback

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 65 / 84

Sullivan 1990

GainFeedbackAmplifier

...

Pre-distortion output level

Pre-distortion gain

Output Signal

Distortion output level

Nonlinear Distortion

Amplifier Feedback Delay

String 1

String N

Distortion output signal often further filtered by an amplifier cabinetfilter, representing speaker cabinet, driver responses, etc.

Page 66: History and Practice of Digital Sound Synthesis

Distortion Guitar Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 66 / 84

(Stanford Sondius Project, ca. 1995)

• Distortion Guitar: (WAV) (MP3)• Amplifier Feedback 1: (WAV) (MP3)• Amplifier Feedback 2: (WAV) (MP3)

Page 67: History and Practice of Digital Sound Synthesis

Commuted Synthesis of Acoustic Strings (1993)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 67 / 84

Trigger Outpute(t) s(t) y(t)

ResonatorStringExcitation

Schematic diagram of a stringed musical instrument.

Trigger OutputResonator StringExcitation

Equivalent diagram in the linear, time-invariant case.

AggregateExcitation

a(t)String

x(t)OutputTrigger

Use of an aggregate excitation given by the convolution of originalexcitation with the resonator impulse response.

Page 68: History and Practice of Digital Sound Synthesis

Commuted Components

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 68 / 84

AggregateExcitation

a(t)String

x(t)OutputTrigger

“Plucked Resonator” driving a String.

y(t)BridgeCoupling

s(t) GuitarBody

AirAbsorption

RoomResponse Output

Possible components of a guitar resonator.

Page 69: History and Practice of Digital Sound Synthesis

Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 69 / 84

Electric Guitar (Pick-Ups and/or Body-Model Added) (StanfordSondius Project → Staccato Systems, Inc. → ADI, ca. 1995)

• Example 1: (WAV) (MP3)• Example 2: (WAV) (MP3)• Example 3: (WAV) (MP3)• Virtual “wah-wah pedal”: (WAV) (MP3)

STK Mandolin

• STK Mandolin 1: (WAV) (MP3)• STK Mandolin 2: (WAV) (MP3)

Page 70: History and Practice of Digital Sound Synthesis

Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 70 / 84

More Recent Acoustic Guitar

• Bach Prelude in E Major: (WAV) (MP3)• soundexamplewavBach silenceLoure in E Major: (WAV) (MP3)

Virtual performance by Dr. Mikael Laurson, Sibelius Institute

Virtual guitar by Helsinki Univ. of Tech., Acoustics Lab3

3http://www.acoustics.hut.fi/

Page 71: History and Practice of Digital Sound Synthesis

Commuted Synthesis of Linearized Violin

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 71 / 84

e(n) x(n)s(n)Amplitude(n)Frequency(n)

x(n)Amplitude(n)Frequency(n)

a(n)

a(n) x(n)Output

Impulse-ResponseTrain

Amplitude(n)Frequency(n)

a)

b)

c)

Output

Output

String

Stringe(n)

String Resonator

Resonator

ImpulseTrain

ImpulseTrain

• Assumes ideal Helmholtz motion of string• Sound Examples (Stanford Sondius project, ca. 1995):

Bass: (WAV) (MP3) Cello: (WAV) (MP3) Viola 1: (WAV) (MP3) Viola 2: (WAV) (MP3)

Violin 1: (WAV) (MP3) Violin 2: (WAV) (MP3) Ensemble: (WAV) (MP3)

Page 72: History and Practice of Digital Sound Synthesis

Commuted Piano Synthesis (1995)

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 72 / 84

Hammer-string interaction pulses (force):

5 10 15 20Time

0.1

0.2

0.3

0.4

0.5

Force

Page 73: History and Practice of Digital Sound Synthesis

Synthesis of Hammer-String Interaction Pulse

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 73 / 84

Time Time

LowpassFilter

Impulse Impulse Response

• Faster collisions correspond to narrower pulses(nonlinear filter )

• For a given velocity, filter is linear time-invariant

• Piano is “linearized” for each hammer velocity

Page 74: History and Practice of Digital Sound Synthesis

Multiple Hammer-String Interaction Pulses

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 74 / 84

Superimpose several individual pulses:

Impulse 1 LPF1

LPF2

LPF3

+Impulse 2

Impulse 3

StringInput

Time

Force

0

δ1

δ2

δ3

Page 75: History and Practice of Digital Sound Synthesis

Multiple Hammer-String Interaction Pulses

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 74 / 84

Superimpose several individual pulses:

Impulse 1 LPF1

LPF2

LPF3

+Impulse 2

Impulse 3

StringInput

Time

Force

0

δ1

δ2

δ3

As impulse amplitude grows (faster hammer strike), output pulsesbecome taller and thinner, showing less overlap.

Page 76: History and Practice of Digital Sound Synthesis

Complete Piano Model

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 75 / 84

Natural Ordering:

LPF1

LPF2

LPF3

+ StringOutput

Sound Board& Enclosure

TappedDelayLine

δ1

δ2

δ3

ImpulseGener-

ator

δ1

vc

Trigger

Commuted Ordering:

LPF1

LPF2

LPF3

+ String OutputTappedDelayLineTrigger

vc

Sound Board& Enclosure

Impulse Response

• Soundboard and enclosure are commuted• Only need a stored recording of their impulse response• An enormous digital filter is otherwise required

Page 77: History and Practice of Digital Sound Synthesis

Piano and Harpsichord Sound Examples

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 76 / 84

(Stanford Sondius Project, ca. 1995)

• Piano: (WAV) (MP3)• Harpsichord 1: (WAV) (MP3)• Harpsichord 2: (WAV) (MP3)

Page 78: History and Practice of Digital Sound Synthesis

More Recent Harpsichord Example

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 77 / 84

• Harpsichord Soundboard Hammer-Response: (WAV) (MP3)• Musical Commuted Harpsichord Example: (WAV) (MP3)

Reference:

“Sound Synthesis of the Harpsichord Using a ComputationallyEfficient Physical Model”,

by Vesa Valimaki, Henri Penttinen, Jonte Knif, Mikael Laurson,and Cumhur Erkut

JASP-2004

Google search: Harpsichord Sound Synthesis

Page 79: History and Practice of Digital Sound Synthesis

Physical Modeling in Audio Coding

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 78 / 84

Spectral modeling synthesis is finding application in audio coding.Can physical modeling synthesis be used as well?

• MPEG-4/SAOL already supports essentially all sound synthesismethods

• Ability to encode sounds automatically is limited

Codebook-Excited Linear Prediction (CELP) is a successfulsource-filter model (not quite physical)

There are many isolated examples of model-fitting torecorded data

Good model-based denoising results have been obtained Coder problem much harder when many sources are mixed

Page 80: History and Practice of Digital Sound Synthesis

Best Known Model-Based Audio Coders

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 79 / 84

A “cover band” can put together a very convincing facsimile ofpopular music performance

JOS high-school band “Bittersweet”

Page 81: History and Practice of Digital Sound Synthesis

Future Physical Modeling in Audio Coding?

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

• KL Music

• Digital Waveguide

• Signal Scattering

• Plucked String

• Struck String

• Karplus Strong

• EKS Algorithm

• Clarinet

• Wind Examples

• Bowed Strings

• Distortion Guitar

• Acoustic Strings

• Sound Examples

• Linearized Violin

• Commuted Piano

• Pulse Synthesis

• Complete Piano

• Sound Examples

• Phy Audio Coding

• Phy Audio Coding?

Summary

Julius Smith AES-2006 Heyser Lecture – 80 / 84

A “Cover Band” Approach to Model-Based Audio Coding:

1. Recognize individual “audio streams” in a mix (CASA)(“I hear a trap set, electric bass, Fender Rhodes, and a strat”)

2. For each stream, calibrate its model heuristically(“Here is what I hear the bass part doing: ...”)

3. Fine-tune the synthetic mix to the real mix(joint “maximum likelihood estimation”)

Features of “Cover-Band Coding” (CBC):

• The “playing experience” of each “virtual performer” preventsartifacts — “musically unreasonable” parameters are madeunlikely (“Bayesian priors”)

• An incorrect instrument must “imitate” its assigned stream• New arrangements can be synthesized by deliberately choosing

a new ensemble!

Page 82: History and Practice of Digital Sound Synthesis

Summary

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 81 / 84

Page 83: History and Practice of Digital Sound Synthesis

Summary

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Julius Smith AES-2006 Heyser Lecture – 82 / 84

We have reviewed a “CCRMA-centric slice” of the history of digitalsound synthesis (usually starting with results from Bell Labs):

• Wavetable (one period)• Subtractive• Additive• FM• Sampling• Spectral Modeling• Physical Modeling (more in tomorrow’s 4:30 PM masterclass)• Connections to audio coding

Page 84: History and Practice of Digital Sound Synthesis

Sound Acknowledgment

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Sound Acknowledgment

Julius Smith AES-2006 Heyser Lecture – 83 / 84

Page 85: History and Practice of Digital Sound Synthesis

Sound Acknowledgment

Overview

Early Digital Synthesis

Spectral Modeling

Physical Modeling

Summary

Sound Acknowledgment

Julius Smith AES-2006 Heyser Lecture – 84 / 84

Thanks to Emu / Creative Labs for providing a superb-qualityexternal D/A converter for this talk (an E-Mu 0404/USB)