YOU ARE DOWNLOADING DOCUMENT

Please tick the box to continue:

Transcript
  • Sound Synthesis and Processing

    Xavier Serra

    Music Technology Group

    Universitat Pompeu Fabra, Barcelona

  • Xavier Serra - 2008

    Index

    Tradition of sound synthesis

    Musique Concrte, Granular synthesis, Sampling

    Spectral sound processing

    Mosaicing, Concatenative synthesis

    Gesture-based synthesis

  • Xavier Serra - 2008

    Tradition of sound synthesis

  • Xavier Serra - 2008

    4

  • Xavier Serra - 2008

  • Xavier Serra - 2008

    graphical interface for Tbone (Perry Cook)

    Synthesis with physical models

  • Xavier Serra - 2008

    Digital implementation of an acoustic system

  • Xavier Serra - 2008

    Musique concrte

    Pierre SchaefferPierre Schaeffer, Etude aux chemins de fer (1948)

    A class of electronic art music produced from editing together fragments of natural and industrial sounds.

  • Xavier Serra - 2008

    Granular synthesis

    All sound is an integration of grains, of elementary

    sonic particles, of sonic

    quanta Xenakis (1971).

    Helmuths example

  • Xavier Serra - 2008

    Sampling

    Fairlight (1980)Pioneered two innovations that transformed music making, namely sampling and sequencing.

  • Xavier Serra - 2008

    Spectral processing

    Transform. Spectral

    Synthesis Fourier

    Analysis Original Sound

    TransformedSound

    Original Spectrum

    TransformedSpectrum

    Feature Extraction

    Feature Addition

    Original Feature

    Transformed Feature

    Spectral Analysis

  • 12

    ( ) ( )=

    ++=N

    kkkk tfAAtx

    10 2cos pi

    =

    ++=N

    k

    tfjktfjk kk eX

    eX

    X1

    2*

    20 22

    pipi

    wherewhere kjkk eAX

    =

    ( ){ }

    +=

    +=

    =

    =

    +

    N

    k

    tfjjk

    N

    k

    tfjk

    kk

    kk

    eeAA

    eAA

    1

    20

    1

    20

    Re

    Re

    pi

    pi

    Spectral representation

  • 13

    ( ) ( ) ( ) ( ) ( ){ },,,,,,,,,0, 2*22122211*12111210 fXfXfXfXX A spectrum is a sequence of pairs of numbers:A spectrum is a sequence of pairs of numbers:

    ( ) ( ) ( )2500cos83200cos1410 pipipipi +++= tttxThe spectrum of:

  • 14

    Spectral analysis:Spectral analysis:

  • 15

    Short-Time Fourier Transform (STFT)Short-Time Fourier Transform (STFT) Sinusoidal ModelSinusoidal Model Sinusoidal plus Residual ModelSinusoidal plus Residual Model Higher Level AttributesHigher Level Attributes

    Spectral models

  • 16

    soundsound

    additiveadditivesynthesissynthesis

    sinusoidalsinusoidalcomponentcomponent

    **windowwindowgenerationgeneration

    smoothingsmoothingwindowwindow

    FFTFFTphasephasespectrumspectrum

    magnitudemagnitudespectrumspectrum

    spectralspectralapproximationapproximation

    residualresidualspectral dataspectral data

    FFTFFT

    magnitudemagnitudespectrumspectrum

    phasephasespectrumspectrum

    peakpeakdetectiondetection peakpeak

    datadata

    pitchpitchdetectiondetection

    peakpeakdatadata

    F0F0frequencyfrequency

    amplitudeamplitudecorrectioncorrection

    peakpeakcontinuationcontinuation

    sine frequenciessine frequenciessine magnitudessine magnitudessine phasessine phases

    **

    windowwindowgenerationgeneration

    smoothingsmoothingwindowwindow

    --residualresidualcomponentcomponent

    Spectral analysis

  • 17

    sine datasine data

    spectral dataspectral data

    amplitude amplitude andand

    frequencyfrequencydetrendingdetrending

    attack,attack,steady-state,steady-state,and decayand decaydetectiondetection

    inflexioninflexionpointspoints

    spectral shapespectral shapeextractionextraction

    spectralspectralshapeshape

    amplitude normalizationamplitude normalizationandand

    amplitude envelopeamplitude envelopeextractionextraction

    amplitude andamplitude andamplitude amplitude envelopeenvelope

    frequency normalizationfrequency normalizationandand

    pitch envelopepitch envelopeextractionextraction

    pitch andpitch andpitch envelopepitch envelope

    loop pointsloop pointsselectionselection

    residualresidualsine datasine data

    residualresidualspectral dataspectral data

    loop pointsloop points

    vibratovibratoextractionextraction

    vibratovibratodatadata

    Feature analysis

  • 18

    IFFTIFFT++complexcomplexspectrumspectrum

    sinesinefrequenciesfrequencies

    sinesinemagnitudesmagnitudes

    sinesinephasesphases

    spectral dataspectral data

    spectral spectral sinesine

    generationgeneration

    magnitudemagnitudespectrumspectrum

    phasephasespectrumspectrum

    polar to polar to rectangularrectangularconversionconversion

    spectralspectralresidualresidual

    generationgeneration

    magnitudemagnitudespectrumspectrum

    phasephasespectrumspectrum

    **

    windowwindowgenerationgeneration

    soundsound

    synthesissynthesiswindowwindow

    **

    spectral dataspectral data(ex: reverberation)(ex: reverberation)

    polar to polar to rectangularrectangularconversionconversion

    Spectral synthesis

  • Xavier Serra - 2008

    Spectral transformations

    Original Sound

    Transformation

    Amp

    f fi-1 fi fi+1 fi+2

    desired timbre envelope

  • Xavier Serra - 2008

    Sound collage

    John Oswald

    A phonograph in the hands of a hip hop/scratch artist produces sounds which are unique and not reproduced - the record player becomes a musical instrument. A sampler is simultaneously a documenting device and a creative device, in effect reducing a distinction manifested by copyright.

    no official distributor/not for sale 1989

  • Xavier Serra - 2008

    Photo mosaicing

    Example of photomosaic made out of hundreds of portraits of Americans who have died at war in Iraq during the last few years.

  • Xavier Serra - 2008

    Audio mosaicing

    (Schwarz, 2007)

    Tristan Jehan, 2005

  • Xavier Serra - 2008

    Concatenative synthesis (1 of 2)

    ( ) ( )11

    1 2 22 1 1

    n n

    n n

    i ii i n nn n

    f transp f transpt

    i t pitch transp

    pi

    pi

    + =

    + +

    2 c

    syncn n

    tpitch transp

    pi

    =

    PHASE TIMBRE

    stretchedspectrum

    1-SSIntp

    SPP Spectrum n-i

    differentialenvelope

    Spectral Shape

    Interpolation Factor

    (SSIntp)

    1Right EpRLeft EpR

    Stretch Stretch

    Spectral ShapeInterpolation Factor

    Frame n-k Frame n-1

    Interpolation zonek frames

    Frame n

    f

    f

    F0n-1 F1 n-1 F2 n-1 F3 n-1

    F0 n-1

    F1 n-1

    F2 n-1

    F3 n-1

    EpR frequency mapping f

    f

    F0n-1 F1 n-1 F2 n-1 F3 n-1

    F0 n-1

    F1 n-1

    F2 n-1

    F3 n-1

    EpR frequency mapping

    Timbre Envelope

    stretchedspectrum

    1-SSIntp

    SPP Spectrum n-i

    differentialenvelope

    Spectral Shape

    Interpolation Factor

    (SSIntp)

    1Right EpRLeft EpR

    Stretch Stretch

    Spectral ShapeInterpolation Factor

    Frame n-k Frame n-1

    Interpolation zonek frames

    Frame n

    f

    f

    F0n-1 F1 n-1 F2 n-1 F3 n-1

    F0 n-1

    F1 n-1

    F2 n-1

    F3 n-1

    EpR frequency mapping f

    f

    F0n-1 F1 n-1 F2 n-1 F3 n-1

    F0 n-1

    F1 n-1

    F2 n-1

    F3 n-1

    EpR frequency mapping

    Timbre Envelope

  • Xavier Serra - 2008

    Concatenative synthesis (2 of 2)

    Apossible sounds produced by the instrumentBsounds produced by the performer playing the instrument

    recorded audio samples

    Instrument sonic space

    performance trajectory generator

    performance DB

    sound rendering

    Performance trajectory

    performer model

    Performance Score

    Sound

    Synthesizer diagram

    Vocaloid example

  • Xavier Serra - 2008

    Synthesis based on gestures

    gestures generatedby the system

  • Violin gesture modeling


Related Documents