Nov 30, Fall 2006 IAT 410 1 Audio Sound Audio synthesis
Dec 20, 2015
Nov 30, Fall 2006 IAT 410 1
Audio
SoundAudio synthesis
Nov 30, Fall 2006 IAT 410 2
Audio Perception
Sound: Pressure waves in frequencies between 50Herz - 22,000Herz
Lower frequencies more felt by the whole body than heard
Sounds can be perceived as coming from a location– Not terribly accurate– Cone of confusion
Nov 30, Fall 2006 IAT 410 3
Cone of confusion– Cone-shaped zones in front of and
behind head 3D Audio cues:
– Interaural Time Difference– Interaural Intensity Difference– Pinnae filtering– Body filtering
3D Audio Perception
Nov 30, Fall 2006 IAT 410 4
3D Audio Perception
Goal for 3D sound is “Spatialization” The sense that the
– Sound originates outside your head– Sound has a direction
Interaural Time Difference– The more extremely left or right, the greater
the difference– Time difference < 5ms
Nov 30, Fall 2006 IAT 410 5
3D Audio Perception
Interaural Intensity Difference– Head absorbs and reflects sound energy– The first ear to get sound gets loudest
sound– “Head Shadow”
Cone of confusion:– Time difference too small to detect– Intensity is similar in both ears
Nov 30, Fall 2006 IAT 410 6
Pinnae Filtering
Outer ear (Pinna) shape filters sound based on its direction
Childhood learning trains brain to associate filtering effects with direction
Unique per person Record directional white noise
– Microphone in ear canal– Sounds from speakers located about head
Nov 30, Fall 2006 IAT 410 7
Pinnae Filtering
A “Generic” Pinna can be simulated Record directional white noise
received by dummy head Body filtering
– Reflection and absorption– Included in Pinna model
Nov 30, Fall 2006 IAT 410 8
Head-Related Transfer Function
HRTF is the general term Transformation of “real” sound to
spatialized sound Best delivered by earphones
Nov 30, Fall 2006 IAT 410 9
Environmental Effects
Sound exists in an environment– Bounces off objects– Is absorbed by objects
Simple effects– Reverb: Simulate the environmental echo
• Echo is the attenuated signal• Gives a richer room-like feeling• Larger room has longer time delay
Nov 30, Fall 2006 IAT 410 10
Audio signals
Nyquist limit:– Must sample signal at least twice as
frequently as highest reproducible frequency
– Audio: 44.1KHz (CD)– 22KHz– 11KHz (Analog AM Radio)– 8KHz (Telephone)
Nov 30, Fall 2006 IAT 410 11
Audio - Digital Implications
44,100 Hz– 44,100 Samples/sec– 16-bit samples– Stereo– 172KBytes/sec
Specialized hardware - Sound card
Nov 30, Fall 2006 IAT 410 12
Reproduction
Sampling– Record sounds by whatever means
Synthesis– Analog Synthesis– FM Synthesis– Wavetable Synthesis
Nov 30, Fall 2006 IAT 410 13
Control
MIDI - Musical Instrument Digital Interface
Developed to control music synthesizers– Details of synthesis are controlled by
synthesizer MIDI data
– Sets synthesis parameters– Sets music sequence
Nov 30, Fall 2006 IAT 410 14
Synthesis
Analog Synthesis– Simple sum of frequencies– Select from a palette of source
frequencies– Sum of frequencies is filtered
FM– One frequency is controlled by another
Wavetable– Digitize audio signals
Nov 30, Fall 2006 IAT 410 15
Analog Synthesis
Fourier’s observation– Any signal can be created as the sum of
sine waves– Square wave: Infinite sum– f + 2f + 4f…
Synthesizer:– Collection of
oscillators
Nov 30, Fall 2006 IAT 410 16
Frequency Domain
A perfectly periodic signal plotted in the frequency domain
(Time domain plot)
Nov 30, Fall 2006 IAT 410 17
Spectrum
Spectrum represents the set of frequencies present in the signal
Nov 30, Fall 2006 IAT 410 18
Filters
Eliminate part of the signal by removing certain frequencies
Analog filters don’t have these “square” response shapes
Band pass – Bandwidth
Nov 30, Fall 2006 IAT 410 19
FM Synthesis Modulate the frequency of a wave
Carrier frequency is modulated by Modulator signal
Nov 30, Fall 2006 IAT 410 20
FM Synthesis for Synthesizers
The greater the Modulator amplitude, the greater the Carrier frequency variation– Higher Carrier bandwidth
DX: Carrier and Modulator are “musically-tuned frequency”– Depends on the note you are playing – Controls the harmonic content of a note
Nov 30, Fall 2006 IAT 410 21
Wavetable Synthesis
Collect a sample of the real sound Issues:
– Reduce memory load by looping sample– Shift pitch instead of sampling each
individual note– Apply interpolation techniques to make
pitch shifting work right
Nov 30, Fall 2006 IAT 410 22
Audio Path
Raw Sound(Sample, FM, etc)
Tuned
MIDINote Sequence
Resampling
EnvelopeLoudness Control
Mixing/Combination
Reverb,Environment.Spatialization
Nov 30, Fall 2006 IAT 410 23
Wavetable Synthesis Example
Leyanda (Guitar) Leyanda (CDShaw)
Nov 30, Fall 2006 IAT 410 24
Interactive Sound
Goal– Want to enhance the interactive
experience– Give the user a sense of presence– Add to the emotional content of the
game– Make it more fun
Nov 30, Fall 2006 IAT 410 25
Interactive Sound
Music Sound effects
– Noises– Commentary - Sports
Narrative Conversations
Nov 30, Fall 2006 IAT 410 26
Interactive Problems
Regular music composition has – Beginning– Middle– End
Interactive user control makes this difficult– Some genres have this structure
Nov 30, Fall 2006 IAT 410 27
Interactive Music
Game genres with order– Sports– Racing– Fighting
Semi-Ordered– Puzzle– Adventure
Nov 30, Fall 2006 IAT 410 28
Music Genres
The Infinite Loop– Theme and variation plays forever– Pomp & Circumstance– Diablo
Problems:– 30 second piece repeated over 6 hours!– 720 repetitions!– Diablo example: 12 Repetitions/hour
Nov 30, Fall 2006 IAT 410 29
Repetition Solutions
Make the Dominant theme hard to find– No catchy theme!– Create a variety of textures– Make only transitions stand out
Where repetition is small– Don’t repeat musical phrases
Nov 30, Fall 2006 IAT 410 30
Music Strategies
Play Win or Lose music– Music must be long enough to be
meaningful– Music may be so long that the game
situation changes before completion– Very short music makes little sense
Interrupt current music– Sounds jarring
Nov 30, Fall 2006 IAT 410 31
Modules
Modular chunks– Each segment of the game plays
independently of others– Some thematic relation– Disjointed
Nov 30, Fall 2006 IAT 410 32
Music Strategies
Compose many themes in parallel Switch between themes Connect modular components
together
Nov 30, Fall 2006 IAT 410 33
Analogy: Parallel Trains
N trains of music running in parallel Each train serves an emotional
purpose– Train A: Calm– Train B: Rising Excitement– Train C: Climactic moments– Train D: Falling Excitement
Generally, Train A would be most commonly played
Nov 30, Fall 2006 IAT 410 34
Within Each Musical Train
Each “Car” contains a few bars of music Switch between trains when a “Car” is
complete– Don’t switch in the middle of a “Car”
Simple version:– Each musical phrase ends on last bar of
“Car” Complex:
– Notes at end are carried over to next
Nov 30, Fall 2006 IAT 410 35
Parallel Trains
Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32
Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32
Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32
Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32
Nov 30, Fall 2006 IAT 410 36
Parallel Trains: Shuffle Cars
Shuffle cars – Instead of playing cars in order
Problem: Random cars sound like random radio tuning
Must determine – Appropriate car pairings– Reasonable paths
Nov 30, Fall 2006 IAT 410 37
Repetition In Trains
Use repeated phrases carefully Maybe use a statistical tool to
analyze paths– Bayesian nets
Endings:– Use transitions as an opportunity to
“End” – Use next Car to “Begin” new series
Nov 30, Fall 2006 IAT 410 38
Composing a Train
Create a piece with all layers– Piece can probably survive a layer or two
removed– Variation = piece with layer removed
Be careful with prominent instruments– Fallback: Use instruments with similar
acoustical properties• Piano, Organ, Woodwinds• No Trumpets, Drums or Screaming guitar!