Top Banner
Expressive Sound Synthesis For Animation Cécile Picard-Limpens University of Nice/Sophia-Antipolis École Doctorale STIC REVES INRIA Sophia-Antipolis, France Advisors : George Drettakis, INRIA Sophia Antipolis (Reves) François Faure, INRIA Rhône-Alpes (Evasion) Nicolas Tsingos, DOLBY Laboratories, CA, USA Defense for Ph.D. in Computer Science C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 1
65

Ph.D. Defense: Expressive Sound Synthesis for Animation

May 24, 2015

Download

Technology

Cecile P-L

The main objective of this thesis is to provide tools for an expressive and real-time synthesis of sounds resulting from physical interactions of various objects in a 3D virtual environment. Indeed, these sounds, such as collisions sounds or sounds from continuous interaction between surfaces, are
difficult to create in a pre-production process since they are highly dynamic and vary drastically depending on the interaction and objects. To achieve this goal, two approaches are proposed; the first one is based on simulation of physical phenomena responsible for sound production, the second one is based on the processing of a recordings database.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Ph.D. Defense: Expressive Sound Synthesis for Animation

tugraz

Expressive Sound SynthesisFor Animation

Cécile Picard-Limpens

University of Nice/Sophia-AntipolisÉcole Doctorale STIC

REVES INRIA Sophia-Antipolis, FranceAdvisors: George Drettakis, INRIA Sophia Antipolis (Reves)

François Faure, INRIA Rhône-Alpes (Evasion)Nicolas Tsingos, DOLBY Laboratories, CA, USA

Defense for Ph.D. in Computer Science

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation1

Page 2: Ph.D. Defense: Expressive Sound Synthesis for Animation

tugraz

Outline

1 Sound and Virtuality

2 Physics-Based Sound SynthesisContact ModelingResonator Modeling

3 Example-Based SynthesisFlexible Sound Synthesis

4 Perspectives on a Hybrid ModelMotivation and Application

5 Conclusion and DiscussionContributionsExtensions and Applications

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation2

Page 3: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

General Background

Motivation

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Sound Renderingfor Virtual Reality and Games

Interactive Audio Rendering

(R. Vantielcke - WipeoutHD on Playstation 3)

Traditional ApproachPre-Recordings Triggered

+ : Easy to implement– : Repetitive audio, discrepancies, lack of flexibility

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation3

Page 4: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

General Background

Motivation

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Sound Renderingfor Virtual Reality and Games

Interactive Audio Rendering

(R. Vantielcke - WipeoutHD on Playstation 3)

Traditional ApproachPre-Recordings Triggered

+ : Easy to implement– : Repetitive audio, discrepancies, lack of flexibility

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation3

Page 5: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

General Background

Motivation

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

From Playback of Samplesto Synthesis

Digital Sound SynthesisSource modeling ←↩Sound propagation, Sound reception

(ArtiSynth)

TechniquesRigid body simulationFinite Element Method (FEM)

Physical Sound Simulation+ : Physical approach, easy parametrization,

Low memory usage– : Preprocess computation,

Interface between physics and sound system

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation4

Page 6: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

General Background

Motivation

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

From Playback of Samplesto Synthesis

Digital Sound SynthesisSource modeling ←↩Sound propagation, Sound reception

(ArtiSynth)

TechniquesRigid body simulationFinite Element Method (FEM)

Physical Sound Simulation+ : Physical approach, easy parametrization,

Low memory usage– : Preprocess computation,

Interface between physics and sound system

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation4

Page 7: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

General Background

Motivation

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Controlling the Sound SimulationChallenges

Sound Coherent With Visuals

Unpredictable character of soundsReal-time sound synthesis

Parametrization and Expressiveness

Control and interactivityAuthoring

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation5

Page 8: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

General Background

Motivation

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Our ContributionThree Research Axes

Physics-Based Sound synthesisContact modelingResonator modeling

Example-Based Sound SynthesisAutomatic analysis of pre-recordingsFlexible synthesis for physics-driven animation

Perspectives on a Hybrid Model

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation6

Page 9: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

General Background

Motivation

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Our ContributionThree Research Axes

Physics-Based Sound synthesisContact modelingResonator modeling

Example-Based Sound SynthesisAutomatic analysis of pre-recordingsFlexible synthesis for physics-driven animation

Perspectives on a Hybrid Model

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation6

Page 10: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

General Background

Motivation

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Our ContributionThree Research Axes

Physics-Based Sound synthesisContact modelingResonator modeling

Example-Based Sound SynthesisAutomatic analysis of pre-recordingsFlexible synthesis for physics-driven animation

Perspectives on a Hybrid Model

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation6

Page 11: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Overview

1 Sound and Virtuality

2 Physics-Based Sound SynthesisContact ModelingResonator Modeling

3 Example-Based SynthesisFlexible Sound Synthesis

4 Perspectives on a Hybrid ModelMotivation and Application

5 Conclusion and DiscussionContributionsExtensions and Applications

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation7

Page 12: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Sound from Contacts

DichotomyImpactsContinuous contacts

Two Schemes for Contact Force Modelling

Additive synthesis

Feed-forward scheme[van den Doel et al . ′01]

Bristle model

Direct computation of contact forces[Avanzini et al . ′02]

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation8

Page 13: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Contact Modeling

What Are The Current Limitationsfor Continuous Contacts?

Rate for physics engine reportNo geometric details when using visual texturesAuthoring and control are challenging

HOW Can We Solve Them?By extracting

Excitation profiles from visual textureswithAdaptive resolution[Picard et al ., VRIPHYS ′08]

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation9

Page 14: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Contact Modeling

What Are The Current Limitationsfor Continuous Contacts?

Rate for physics engine reportNo geometric details when using visual texturesAuthoring and control are challenging

HOW Can We Solve Them?By extracting

Excitation profiles from visual textureswithAdaptive resolution[Picard et al ., VRIPHYS ′08]

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation9

Page 15: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Method for Impact Sounds

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation10

Page 16: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Method for Continuous Contact SoundsExtraction of Excitation Profiles

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation11

Page 17: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Synthesis of Excitation ProfilesFor the Audio Force Modelling

Technique

Extraction from the visual texture imageRe-sampling along the trajectoryof the contact interaction (60Hz vs 44kHz)

Based on the Complexity of the Histogram

Simple texture image:Gradient of the image intensity

Complex texture image:Isocurves of constant brightness (isophotes)

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation12

Page 18: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Complex TexturesCoding the Excitation Profiles

Isophotes = Large amount of dataHow Can We Lighten the Info?

By Coding the Excitation Profiles= Main Features + Noise Part

= +Noise Part: Statistical approximation

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation13

Page 19: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Real-Time Audio ManagementA Flexible Audio Pipeline

Simulations Driven by Ageia’s PhysX (now NVIDIA)

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation14

Page 20: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Audio Texture SynthesisA Solution for Interactive Simulations

A Sound in Coherence with Visuals

Flexible Resolution

Adapted to Procedural Generation

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation15

Page 21: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Overview

1 Sound and Virtuality

2 Physics-Based Sound SynthesisContact ModelingResonator Modeling

3 Example-Based SynthesisFlexible Sound Synthesis

4 Perspectives on a Hybrid ModelMotivation and Application

5 Conclusion and DiscussionContributionsExtensions and Applications

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation16

Page 22: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Vibration ModelsModal Analysis

Generating Sounds Based on Physics SimulationIn computer musics[Iovino et al . ′97, Cook ′02]

In computer graphics[Van Den Doel ′01, O′Brien et al . ′02]

Improvements for Interactive Sound RenderingModal parameter tracking[Maxwell et al . ′07]

Frequency content sparsity[Bonneel et al .′08]

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation17

Page 23: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Vibration ModelsModal Analysis

1 Get a Sounding Object and its Geometry

2 Construct the FEM (ex: Tetrahedral Mesh)3 Apply Newton Second Law to DOF

Md + Cd + Kd = f (1)

4 Eigendecomposition ⇒ Modal Parameters

M = LL−T ; L−1KL−T = V ΛV T (2)where V = matrix of eigenvectors

Λ = diagonal matrix of eigenvalues

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation18

Page 24: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Vibration ModelsModal Analysis

In Real-time:Modal synthesis

s(t) =∑1

nai sin(wi t)e−di t (3)

Control for vibration models

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation19

Page 25: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Vibration ModelsModal Analysis

What AreThe Current Limitations?

Meshing is difficultNo real control on the FEM resolutionNo clear interface between physics and audio

HOW Can We Solve Them?By proposing

A robust and multi-scale modal analysiswhich isCoherent with the physics simulation[Picard et al ., DAFx ′09]

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation20

Page 26: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Vibration ModelsModal Analysis

What AreThe Current Limitations?

Meshing is difficultNo real control on the FEM resolutionNo clear interface between physics and audio

HOW Can We Solve Them?By proposing

A robust and multi-scale modal analysiswhich isCoherent with the physics simulation[Picard et al ., DAFx ′09]

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation20

Page 27: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Our Deformation Model

Inspired from Work by Nesme et al.[Nesme et al .′06]

TechniqueMerged voxels used as Hexahedral Finite Elements

Implementation with the Sofa Framework

Validation of the ModelTests on a metal cube

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation21

Page 28: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Robustness

Robust Even for Non-Manifold Geometries

Material: Aluminium

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation22

Page 29: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Multi-Scale for Efficient Memory Usage

A Squirrel in Pine Wood

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation23

Page 30: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Multi-Scale for Efficient Memory Usage

A Squirrel in Pine Wood: Different FE resolutions

3x3x3 4x4x4 8x8x8 9x9x9

Frequency Content = f (Hexahedral FE Resolution)

Higher resolution models

Frequency centroid shift

Convergence of the frequency content

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation24

Page 31: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Comparison with Classical Approach

Sounding Bowl - Material: Aluminium

Classical Approach Our Approach(816 modes) (75 modes)

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation25

Page 32: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

A Robust and Multi-Scale Modal AnalysisA Solution for Sound Synthesis

Realistic

Adapted to Non-Manifold Geometries

Resources Flexibility

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation26

Page 33: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Overview

1 Sound and Virtuality

2 Physics-Based Sound SynthesisContact ModelingResonator Modeling

3 Example-Based SynthesisFlexible Sound Synthesis

4 Perspectives on a Hybrid ModelMotivation and Application

5 Conclusion and DiscussionContributionsExtensions and Applications

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation27

Page 34: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Implementation of Signal-Based Models

[Dobashi et al.′03

]

Concatenative Synthesis[Roads ′91, Schwarz ′06]

Sound Textures Based on Physics[Cook ′99][Dobashi et al . ′03, Zheng et al . ′09]

[Cook ′99

]Authoring and Interactive Control[Cook ′02]

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation28

Page 35: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Implementation of Signal-Based Models

What AreThe Current Limitations?

Processing is not genericParametrizing is difficult

HOW Can We Solve Them?By

Retargetting example soundsTo physics-driven animation[Picard et al ., AES ′09]

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation29

Page 36: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Implementation of Signal-Based Models

What AreThe Current Limitations?

Processing is not genericParametrizing is difficult

HOW Can We Solve Them?By

Retargetting example soundsTo physics-driven animation[Picard et al ., AES ′09]

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation29

Page 37: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Our Approach

SINUSOIDAL

1 DICTIONARY OF AUDIO GRAINS Impulsive / Continuous

2 CORRELATION PATTERNS

R E T A R G E T T I N G T O A N I M A T I O N

AUDIO RENDERER VIDEO RENDERER

PREPROCESSING

INTERACTIVE

OBJECT GEOMETRYVIRTUAL ENVIRONMENT

RIGID-BODYSIMULATION

BUILD COLLISIONSTRUCTURES

DEFINE PROCEDURES

ANIMATION WITH AUDIO

AUDIORECORDING

Our Contributions

TRANSIENT

Amplitude

Time

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation30

Page 38: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Preprocess: A Generic Analysis

Impulsive and Continuous ContactsSpectral Modeling Synthesis (SMS) [Serra ′97]

Automatic Extraction of Audio GrainsDictionary: Impulsive/Continuous

Generation of Correlation Patternsbetween original recordings and audio grains

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation31

Page 39: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

On-Line: Flexible Sound Synthesis

Resynthesis of the Original RecordingsCandidate grains: max. correlation amplitude

Interactive Physics-Driven AnimationsPhysics Info for Retargetting

Contact type: impulsive or continuous?Penetration force and relative velocity

Flexible Audio Shading ApproachAdditional, User-defined Resynthesis Schemes

Spectral domain adaptation/modification

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation32

Page 40: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

On-Line: Flexible Sound Synthesis

Resynthesis of the Original RecordingsCandidate grains: max. correlation amplitude

Interactive Physics-Driven AnimationsPhysics Info for Retargetting

Contact type: impulsive or continuous?Penetration force and relative velocity

Flexible Audio Shading ApproachAdditional, User-defined Resynthesis Schemes

Spectral domain adaptation/modification

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation32

Page 41: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

On-Line: Flexible Sound Synthesis

Resynthesis of the Original RecordingsCandidate grains: max. correlation amplitude

Interactive Physics-Driven AnimationsPhysics Info for Retargetting

Contact type: impulsive or continuous?Penetration force and relative velocity

Flexible Audio Shading ApproachAdditional, User-defined Resynthesis Schemes

Spectral domain adaptation/modification

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation32

Page 42: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Resynthesis of the Original Recordings

94 recordings (14.6Mb)≈ 5000 grains + 94 Correlation Patterns (20% Gain)

Breaking Glass

Shooting Gun

Rolling

Additional Material:http://www-sop.inria.fr/members/Cecile.Picard/"‘Supplemental AES"’

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation33

Page 43: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Flexible Audio Shading Approach

Easy Implementation of Time-Scaling

Faster RollingSlower Breaking

Synthesis of An Infinity Similar Audio Eventsby varying the audio content

Rythmic pattern from Breaking StoneNew material content: stone and gun

Rythmic pattern from Breaking GlassNew material content: ceramic

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation34

Page 44: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Interactive Physics-Driven Animations

Simulations Driven by Sofa Framework

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation35

Page 45: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Flexible SoundSynthesis

Retargetting ExampleSounds

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Retargetting Example SoundsA Solution for Interactive Simulations

Variety

Adapted to Scenarios

Small Memory FootprintReal-Time Rendering

An attractive solution for industrial applications(Eden Games, an ATARI game studio)

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation36

Page 46: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Motivation

A Hybrid Model forFracture Events

Conclusion andDiscussion

tugraz

Overview

1 Sound and Virtuality

2 Physics-Based Sound SynthesisContact ModelingResonator Modeling

3 Example-Based SynthesisFlexible Sound Synthesis

4 Perspectives on a Hybrid ModelMotivation and Application

5 Conclusion and DiscussionContributionsExtensions and Applications

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation37

Page 47: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Motivation

A Hybrid Model forFracture Events

Conclusion andDiscussion

tugraz

Sound ModelingWhen Nonlinearity Occurs

Problems of Single ModelsVibration models assume linearityExample-based sounds are hard to parametrize

Previous WorkModeling nonlinearities[O′Brien et al . ′01, Chadwick et al . ′09][Cook ′02]

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation38

Page 48: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Motivation

A Hybrid Model forFracture Events

Conclusion andDiscussion

tugraz

Sound ModelingWhen Nonlinearity Occurs

Problems of Single ModelsVibration models assume linearityExample-based sounds are hard to parametrize

Previous WorkModeling nonlinearities[O′Brien et al . ′01, Chadwick et al . ′09][Cook ′02]

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation38

Page 49: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Motivation

A Hybrid Model forFracture Events

Conclusion andDiscussion

tugraz

Fracture Events

BackgroundFrequently occur in virtual environments

Visual rendering[O′Brien et al . ′99, ′02][Parker and O′Brien. ′09]

Sound rendering: Little research[Warren et al . ′84] [Rath et al . ′03]

ChallengesEvent depends on the material involvedDifferents phases emerge from fracture event

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation39

Page 50: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Motivation

A Hybrid Model forFracture Events

Conclusion andDiscussion

tugraz

Fracture Events

BackgroundFrequently occur in virtual environments

Visual rendering[O′Brien et al . ′99, ′02][Parker and O′Brien. ′09]

Sound rendering: Little research[Warren et al . ′84] [Rath et al . ′03]

ChallengesEvent depends on the material involvedDifferents phases emerge from fracture event

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation39

Page 51: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Motivation

A Hybrid Model forFracture Events

Conclusion andDiscussion

tugraz

Parametrization of Our Hybrid Model

Selection CriteriaHybrid model applied when nonlinearity occurs

Techniques

FM synthesis

FM synthesisAudio grains

ParametrizationSmooth transition with vibration modelCoherence inside the hybrid model

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation40

Page 52: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Motivation

A Hybrid Model forFracture Events

Conclusion andDiscussion

tugraz

Discussion

Prospective model

Possible problem: report from the physics engine

Simplicity of the tools allows real-time rendering

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation41

Page 53: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

Overview

1 Sound and Virtuality

2 Physics-Based Sound SynthesisContact ModelingResonator Modeling

3 Example-Based SynthesisFlexible Sound Synthesis

4 Perspectives on a Hybrid ModelMotivation and Application

5 Conclusion and DiscussionContributionsExtensions and Applications

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation42

Page 54: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

Synthesis of Sounds for AnimationDifficulties

Audio-Visual Coherence

Extremely Dynamic Character

Precision of Synthesis

Large Variety of Objects

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation43

Page 55: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

ContributionsAn Overview

Complex Contact Modeling2D visual textures used as roughness mapsAudible and position-dependent variationsDetail-layer mechanisms

Improved Modal Analysis for Resonator ModelingComplex non-manifold geometries can be handledMulti-scale resolutionCoherence between simulation and audio

Flexibility of Sound DesignAudio grains and correlation patternsDynamic retargetting to eventsExtended sound synthesis capabilities

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation44

Page 56: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

ContributionsAn Overview

Complex Contact Modeling2D visual textures used as roughness mapsAudible and position-dependent variationsDetail-layer mechanisms

Improved Modal Analysis for Resonator ModelingComplex non-manifold geometries can be handledMulti-scale resolutionCoherence between simulation and audio

Flexibility of Sound DesignAudio grains and correlation patternsDynamic retargetting to eventsExtended sound synthesis capabilities

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation44

Page 57: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

ContributionsAn Overview

Complex Contact Modeling2D visual textures used as roughness mapsAudible and position-dependent variationsDetail-layer mechanisms

Improved Modal Analysis for Resonator ModelingComplex non-manifold geometries can be handledMulti-scale resolutionCoherence between simulation and audio

Flexibility of Sound DesignAudio grains and correlation patternsDynamic retargetting to eventsExtended sound synthesis capabilities

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation44

Page 58: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

ContributionsPerspectives

A Prospective Hybrid Modelfor Complex Physical Phenomena

Focus on NonlinearityCombination of physically basedand example-based methodsApplication Case: Fracture Events

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation45

Page 59: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

Overview

1 Sound and Virtuality

2 Physics-Based Sound SynthesisContact ModelingResonator Modeling

3 Example-Based SynthesisFlexible Sound Synthesis

4 Perspectives on a Hybrid ModelMotivation and Application

5 Conclusion and DiscussionContributionsExtensions and Applications

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation46

Page 60: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

Promising Directions for Future Work

Complex Contact ModelingTwo interacting texturesSurface-based interactionsAdequate perceptual experiments

Improved Modal Analysis for Resonator ModelingRecent work from [Nesme et al . Siggraph′09]Investigations with GPU for in-line computationComplete integration in a virtual scene

Example-Based TechniqueClustering of similar grainsStatistical analysis of correlation patternsPhysics engine design

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation47

Page 61: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

Promising Directions for Future Work

Complex Contact ModelingTwo interacting texturesSurface-based interactionsAdequate perceptual experiments

Improved Modal Analysis for Resonator ModelingRecent work from [Nesme et al . Siggraph′09]Investigations with GPU for in-line computationComplete integration in a virtual scene

Example-Based TechniqueClustering of similar grainsStatistical analysis of correlation patternsPhysics engine design

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation47

Page 62: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

Promising Directions for Future Work

Complex Contact ModelingTwo interacting texturesSurface-based interactionsAdequate perceptual experiments

Improved Modal Analysis for Resonator ModelingRecent work from [Nesme et al . Siggraph′09]Investigations with GPU for in-line computationComplete integration in a virtual scene

Example-Based TechniqueClustering of similar grainsStatistical analysis of correlation patternsPhysics engine design

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation47

Page 63: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

Promising Directions for Future Work

Hybrid Model for Fracture EventsFracture sound simulation frameworkTracking of relevant physical data

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation48

Page 64: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

Conclusion

New Physically Based Algorithmsfor Sound RenderingFlexibility of Sound ModelingIdeas on an Adequate Hybrid Sound Model

Additional info:http://www-sop.inria.fr/members/Cecile.Picard/

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation49

Page 65: Ph.D. Defense: Expressive Sound Synthesis for Animation

Sound andVirtuality

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

Contributions

Extensions andApplications

tugraz

Acknowledgements

George Drettakis, François Faure,and Nicolas Tsingos

REVES TeamMarie-Paule Cani and the Evasion TeamPaul G. Kry at the McGill University, Montréal

Eden Games, an ATARI game studio, Lyon

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation50