Top Banner
Singing voice synthesis in the context of music technology research Xavier Serra Music Technology Group Universitat Pompeu Fabra, Barcelona http://mtg.upf.edu
35

Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Mar 22, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Singing voice synthesis in the context

of music technology research

Xavier Serra Music Technology Group

Universitat Pompeu Fabra, Barcelona http://mtg.upf.edu

Page 2: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Musical Sound Synthesis

Page 3: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Theremin Lev Termen, 1917

RCA theremin, 1929 Clara Rockmore “Vocalise” by Rachmaninoff

Page 4: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Moog Synthesizer

Walter Carlos “Switched on Bach”, 1968

Moog synthesizer from the 60s

Robert Moog, 1964

Page 5: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

FM synthesis John Chowning, 1972

DX7 by Yamaha, 1983

Page 6: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Waveguide synthesis Julius Smith, 1985

VL1 by Yamaha, 1994

Page 7: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Sampling

Fairlight, 1980

Page 8: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

New musical interfaces

Reactable, 2005

Page 9: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Singing Voice Synthesis

Page 10: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Physical model from 50s

Max Mathews, 1965

Kelly, Lochbaum and Mathews, 1961 “Daisy”

Page 11: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Linear Predictive Coding

Charles Dodge “The days ahead”, 1975

Page 12: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

FOF synthesis Xavier Rodet, 1977

Aria “Queen of the Night” from the Magic Flute by Mozart, 1979

Page 13: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

SinSy Keiichi Tokuda et al., 2009

Page 14: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Vocaloid and Hatsune Miku

Page 15: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Sample editing

Harmonic + Stochastic analysis

sines

residual

Timbre (Resonances, Excitation)

Stationary Note attack Note release Note to note Articulation Vibrato

Excitation + Resonance

model estimation

Database Creation

Page 16: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

residual

1/F

Flat residual excitation

Flat harmonic excitation

EpR spectral Amplitude

Filtering

Filtering

EpR spectral Phase

sines

residual

HpS synthesis

Synthesis

Demo from 2001

Page 17: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Yamaha, 2004

Page 18: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology
Page 19: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Yamaha, 2006

Page 20: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology
Page 21: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology
Page 22: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology
Page 23: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology
Page 24: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology
Page 25: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology
Page 26: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology
Page 27: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Some challenges in singing voice research

Page 28: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology
Page 29: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Carnatic music

T. M. Krishna

Page 30: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Beijing Opera

Zhao Xiu Jun

Page 31: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Intonation

Page 32: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Melodic Motives

Discovery Induction Extraction

Matching Retrieval

+ Characterization

Transform

N dimensions

Page 33: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Lyrics and music

Chinese is a tonal language

ma ma1 ma2 ma3 ma4 妈 麻 ⻢马 骂 吗

“mum” “hemp” “horse” “to scold” “?”

Hypothesis: performers try to reflect tone contours in the melody in order to make lyrics understandable

Page 34: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Conclusion

•  Musical instruments <--> Voice •  Singing voice synthesis <--> Speech synthesis •  Timbre models --> Musical models

•  Lessons learned from Hatsune Miku

Page 35: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology

Thanks