Sound Synthesis and Processingsmcnetwork.org/files/3-Sound-Synthesis-Processing.pdfSound Synthesis and Processing Xavier Serra ... Vocaloid example. Xavier Serra - 2008 Synthesis based

Sound Synthesis and Processing

Xavier Serra

Music Technology Group

Universitat Pompeu Fabra, Barcelona

Xavier Serra - 2008

Index

Tradition of sound synthesis

Musique Concrète, Granular synthesis, Sampling

Spectral sound processing

Mosaicing, Concatenative synthesis

Gesture-based synthesis

Xavier Serra - 2008

Tradition of sound synthesis

Xavier Serra - 2008

4

Xavier Serra - 2008

Xavier Serra - 2008

graphical interface for Tbone (Perry Cook)

Synthesis with physical models

Xavier Serra - 2008

Digital implementation of an acoustic system

Xavier Serra - 2008

Musique concrète

Pierre SchaefferPierre Schaeffer, “Etude aux chemins de fer” (1948)

A class of electronic art music produced from editing together fragments of natural and industrial sounds.

Xavier Serra - 2008

Granular synthesis

“All sound is an integration of grains, of elementary

sonic particles, of sonic

quanta” Xenakis (1971).

Helmuth’s example

Xavier Serra - 2008

Sampling

Fairlight (1980)Pioneered two innovations that transformed music making, namely sampling and sequencing.

Xavier Serra - 2008

Spectral processing

Transform. Spectral

Synthesis Fourier

Analysis Original Sound

TransformedSound

Original Spectrum

TransformedSpectrum

Feature Extraction

Feature Addition

Original Feature

Transformed Feature

Spectral Analysis

12

( ) ( )∑=

++=N

kkkk tfAAtx

10 2cos φπ

∑=

−

++=N

k

tfjktfjk kk eX

eX

X1

2*

20 22

ππ

wherewhere kjkk eAX φ=

( ){ }

+=

+=

∑

∑

=

=

+

N

k

tfjjk

N

k

tfjk

kk

kk

eeAA

eAA

1

20

1

20

Re

Re

πφ

φπ

Spectral representation

13

( ) ( ) ( ) ( ) ( ){ },,,,,,,,,0, 2*22

1222

11

*12

1112

10 fXfXfXfXX −−

A spectrum is a sequence of pairs of numbers:A spectrum is a sequence of pairs of numbers:

( ) ( ) ( )2500cos83200cos1410 ππππ ++−+= tttxThe spectrum of:

14

Spectral analysis:Spectral analysis:

15

• Short-Time Fourier Transform (STFT)Short-Time Fourier Transform (STFT)

• Sinusoidal ModelSinusoidal Model

• Sinusoidal plus Residual ModelSinusoidal plus Residual Model

• Higher Level AttributesHigher Level Attributes

Spectral models

16

soundsound

additiveadditivesynthesissynthesis

sinusoidalsinusoidalcomponentcomponent

**windowwindowgenerationgeneration

smoothingsmoothingwindowwindow

FFTFFTphasephasespectrumspectrum

magnitudemagnitudespectrumspectrum

spectralspectralapproximationapproximation

residualresidualspectral dataspectral data

FFTFFT

magnitudemagnitudespectrumspectrum

phasephasespectrumspectrum

peakpeakdetectiondetection

peakpeakdatadata

pitchpitchdetectiondetection

peakpeakdatadata

F0F0frequencyfrequency

amplitudeamplitudecorrectioncorrection

peakpeakcontinuationcontinuation

sine frequenciessine frequenciessine magnitudessine magnitudessine phasessine phases

**

windowwindowgenerationgeneration

smoothingsmoothingwindowwindow

--residualresidualcomponentcomponent

Spectral analysis

17

sine datasine data

spectral dataspectral data

amplitude amplitude andand

frequencyfrequencydetrendingdetrending

attack,attack,steady-state,steady-state,and decayand decaydetectiondetection

inflexioninflexionpointspoints

spectral shapespectral shapeextractionextraction

spectralspectralshapeshape

amplitude normalizationamplitude normalizationandand

amplitude envelopeamplitude envelopeextractionextraction

amplitude andamplitude andamplitude amplitude envelopeenvelope

frequency normalizationfrequency normalizationandand

pitch envelopepitch envelopeextractionextraction

pitch andpitch andpitch envelopepitch envelope

loop pointsloop pointsselectionselection

residualresidualsine datasine data

residualresidualspectral dataspectral data

loop pointsloop points

vibratovibratoextractionextraction

vibratovibratodatadata

Feature analysis

18

IFFTIFFT++complexcomplexspectrumspectrum

sinesinefrequenciesfrequencies

sinesinemagnitudesmagnitudes

sinesinephasesphases

spectral dataspectral data

spectral spectral sinesine

generationgeneration

magnitudemagnitudespectrumspectrum

phasephasespectrumspectrum

polar to polar to rectangularrectangularconversionconversion

spectralspectralresidualresidual

generationgeneration

magnitudemagnitudespectrumspectrum

phasephasespectrumspectrum

**

windowwindowgenerationgeneration

soundsound

synthesissynthesiswindowwindow

**

spectral dataspectral data(ex: reverberation)(ex: reverberation)

polar to polar to rectangularrectangularconversionconversion

Spectral synthesis

Xavier Serra - 2008

Spectral transformations

Original Sound

Transformation

Amp

f fi-1 fi fi+1 fi+2

desired timbre envelope

Xavier Serra - 2008

Sound collage

John Oswald

“A phonograph in the hands of a hip hop/scratch artist produces sounds which are unique and not reproduced - the record player becomes a musical instrument. A sampler is simultaneously a documenting device and a creative device, in effect reducing a distinction manifested by copyright.”

no official distributor/not for sale 1989

Xavier Serra - 2008

Photo mosaicing

Example of photomosaic made out of hundreds of portraits of Americans who have died at war in Iraq during the last few years.

Xavier Serra - 2008

Audio mosaicing

(Schwarz, 2007)

Tristan Jehan, 2005

Xavier Serra - 2008

Concatenative synthesis (1 of 2)

( ) ( )

111 2

2

2 1 1

n n

n n

i ii i n nn n

f transp f transpt

i t pitch transp

ϕ π

π

φ −−−

�+ �= − �¿ ∆

+ + ∆ ¿× −

2 ·c

syncn n

tpitch transp

ϕπ

−∆∆ =

PHASE TIMBRE

stretchedspectrum

1-SSIntp

SPP Spectrum n-i

differentialenvelope

Spectral Shape

Interpolation Factor

(SSIntp)

1Right EpRLeft EpR

Stretch Stretch

Spectral ShapeInterpolation Factor

Frame n-k Frame n-1

Interpolation zonek frames

Frame n

f

f

F0n-1 F1 n-1 F2 n-1 F3 n-1

F0 n-1

F1 n-1

F2 n-1

F3 n-1

EpR frequency mapping f

f

F0n-1 F1 n-1 F2 n-1 F3 n-1

F0 n-1

F1 n-1

F2 n-1

F3 n-1

EpR frequency mapping

Timbre Envelope

stretchedspectrum

1-SSIntp

SPP Spectrum n-i

differentialenvelope

Spectral Shape

Interpolation Factor

(SSIntp)

1Right EpRLeft EpR

Stretch Stretch

Spectral ShapeInterpolation Factor

Frame n-k Frame n-1

Interpolation zonek frames

Frame n

f

f

F0n-1 F1 n-1 F2 n-1 F3 n-1

F0 n-1

F1 n-1

F2 n-1

F3 n-1

EpR frequency mapping f

f

F0n-1 F1 n-1 F2 n-1 F3 n-1

F0 n-1

F1 n-1

F2 n-1

F3 n-1

EpR frequency mapping

Timbre Envelope

Xavier Serra - 2008

Concatenative synthesis (2 of 2)

Apossible sounds produced by the instrumentBsounds produced by the performer playing the instrument

recorded audio samples

Instrument sonic space

performance trajectory generator

performance DB

sound rendering

Performance trajectory

performer model

Performance Score

Sound

Synthesizer diagram

Vocaloid example

Xavier Serra - 2008

Synthesis based on gestures

gestures generatedby the system

Violin gesture modeling