Musical Acoustics & Psychoacoustics Donald Byrd 5 Nov. 2007 Copyright © 2006-07, Donald Byrd.

Musical Acoustics & Psychoacoustics

Donald Byrd5 Nov. 2007

Copyright © 2006-07, Donald Byrd

rev. 8 Sep. 2006 2

Review: Representations of Music

• Three basic forms (representations) of music– Audio: most important for most people (general

public)– MIDI files: often best/essential for some musicians,

especially for pop, rock, film/TV– Notation: often best/ essential for musicians (even

amateurs) & music scholars– Essential difference: how much explicit structure

• Music holdings of Library of Congress: over 10M items– Includes over 6M pieces of sheet music and 100K’s of

scores of operas, symphonies, etc.: all notation!

• Differences are profound

1 Sep. 2006 3

Review: Representations of Music & Audio

Audio (e.g., CD, MP3): like speech

Time-stamped Events (e.g., MIDI file): like unformatted text

Music Notation: like text with complex formatting

Digital Audio

Time-stamped Events

Music Notation

1 Sep. 2006 4

Review: Rudiments of Musical Acoustics

• Need some musical acoustics for almost anything in digital audio

• Acoustics: branch of physics that studies sound (of any kind)– Concepts like frequency & amplitude

• Psychoacoustics: study of how sound is perceived; mostly psychology– Concepts like pitch, loudness, timbre

10 Sept. 2006 5

Materials for Studying Audio

• What are interesting sounds really like?– Sine waves, etc. are boring (cf. addsynenv)– Sounds of acoustic instruments are “rich”– Vary in every way: with pitch, loudness, time

• Musical instrument samples• Audacity audio editor

– For Windows, Mac OS 9 and X, Linux– Download from

http://audacity.sourceforge.net/

• Programs in (e.g.) R

10 Sept. 2006 6

Review: Parameters of Musical Sound

• Four basic parameters of a definite-pitched musical note1. pitch: how high or low the sound is: perceptual

analog of frequency

2. duration: how long the note lasts

3. loudness: perceptual analog of amplitude

4. timbre or tone quality

• Above is decreasing order of importance for most Western music

• …and decreasing order of explicitness in CMN (Conventional Music Notation)!

rev. 31 Oct. 2007 7

Scholars (and others) Beware! (1)• Plausible (at the time!) assumptions

– Men have more teeth than women (ancient)– Diseases can’t be transmitted by invisible

organisms (19th century)– Stomach ulcers can’t be caused by

organisms (20th century)

• What you expect & what you see/hear– A recent discovery about kitchen sponges– 1923 New York Times headline: “Dinosaurs

Cavort in Film for Doyle”

rev. 20 Sep. 2006 8

Scholars (and others) Beware! (2)• What you expect & what you hear

– Don & the Kurzweil 250 flute sound– Hammond organ vs. a real pipe organ– Don, a famous musician, & K250 handclaps– Huron on what he “knew” & learned

• R. Moog at Kurzweil & piano touch

rev. 11 Sept. 06 9

Musical Acoustics (1)• Acoustics involves physics• Musical (opposed to architectural, etc.) acoustics

– Frequency (=> pitch)– Amplitude (=> loudness)– Spectrum, envelope, & “other” characteristics => timbre– Partials vs. harmonics

• Psychoacoustics involves psychology/perception– Perceptual coding (for “lossy” compression: MP3, etc.)

• Timbre– Old idea (thru ca. 1960’s?): timbre produced by static

relationships of partials, plus envelope– …but attack often more distinctive than “steady state”!– Rich (interesting) sounds are complex; nothing is static– Time domain (waveform) vs. frequency domain (spectrum,

spectrogram) views

rev. 31 Oct. 07 10

Materials for Studying Acoustics

• Most useful– Piano (preferably a grand)– String instruments (preferably large & bowed)– Drinking glasses filled to different levels– Simple percussion: sleighbell, “rainegg”, etc.– Recording equipment– Audio editor, spectral-analysis program

• Also helpful– Other musical instruments– Tuning fork– Strobe light– Echoic & anechoic chambers

4 April 07 11

Modes of Vibration & Simple Vibrating Systems

• Waves can be transverse or longitudinal– Which is water? Which is sound?

• Simple vibrating systems & waveforms– Tuning fork

• Not many other examples!

– What’s the waveform?

4 April 07 12

Complex Vibrations & Resonance

• Modes of vibration– Air column closed at both ends vibrates at f, 2f, 3f…– …where f = frequency determined by length– Closed at one end, open at other: f, 3f, 5f…– …where f = half the frequency of previous case

• Resonance– “When a system that can vibrate at a certain frequency is

acted on by periodic disturbance at same frequency, vibrations of large amplitude can be produced.” —Backus

– Ex: swing in playground– Ex: string & body of musical instrument– Resonance: tendency of a system to oscillate at maximum

amplitude at a certain frequency.

4 April 07 13

How String Instruments Make Sounds

• Vibrating strings• Waveform is almost a sawtooth => lots of

harmonics!• Bowing

– Rosin to increase friction– Special effects: sul ponticello, col legno, etc.

• Plucking

4 April 07 14

How Woodwinds Make Sounds

• Vibrating air columns• Without reeds: flute

– Acts like open column

• With reeds: oboe, clarinet, bassoon– Clarinet acts like half-open column– Why?– Same length => an octave lower

• Overblowing• What about brass instruments?

4 April 07 15

How Percussion Instruments Make Sounds

• Guess

31 Oct. 07 16

Acoustic Phenomena

• Sympathetic vibration• The harmonic series• Beats

– Chorus effect

• Harmonics (on bowed strings) show:– Modes of vibration– Nodes– Relationship to sul ponticello?

• Architectural acoustics– Resonance (as in I Am Sitting in a Room)– Standing waves

4 April 07 17

Psychocoustic Phenomena

• A matter of psychology (and, to some extent, music theory)

• Basis for auditory illusions– Endless glissando– “Mystery melody”

• Practical application: perceptual coding used in lossy compression (MP3, WMA, etc.)– Masking– Threshhold of audibility

11 Sept. 2006 18

Creating Interesting Sounds (1)

• Periodic, nearly-, & non-periodic signals• Approaches to creating sounds

– Additive synthesis: like painting– Subtractive synthesis: like sculpture– Sampling: like collage– Others: modulation, physical modeling

• Most possible with analog or digital hardware, but analog is limited

rev. 13 Sept. 2006 19

Creating Interesting Sounds (2)

• Cf. “Electronic Music Tutorial” (Ishkur)• Additive Synthesis• Fourier's Theorem

– Any periodic signal => sum of harmonically-related sine waves

• Envelopes: continuous & piecewise linear– Important special case: “ADSR”– Attack, Decay, Sustain, Release

• Phase, interference, & beats– Phase by itself rarely important, but relationships are– Diagrams & demo in CECM Acoustics Primer, Sec. 8– Interference between channels or very close partials

rev. 13 Sept. 06 20

Additive Synthesis & Envelopes• addsynenv does additive synthesis of up to six partials

– Each has arbitrary partial no., starting phase, "ADSR" envelope

– Partial no. can be non-integer => not harmonic– ADSR = Attack/Decay/Sustain/Release (3 breakpoints)– …but addsynenv allows much more complex envelopes– Plays one note with waveform specified by partial nos. &

their envelopes (maybe also phases)– Simultaneously displays “spectrogram” or “sonogram”– …but not waveform– Phase in real world normally has little effect, but can be

critical in recording & digital worlds (e.g., cancellation)• Additive synthesis can’t create aperiodic (non-definite

pitch) sounds• ...or many realistic attacks

11 Sept. 2006 21

Creating Interesting Sounds (3)

• Early CCRMA (Stanford) studies of acoustic instrument sounds– Envelope for each partial with a few segments– Similar to addsynenv

• Subtractive Synthesis• Early (analog: Moog, etc.) synthesizer model• Signal source: sine/square/sawtooth (for

additive)• …or noise & filters (for subtractive)• LFO to add vibrato, tremolo, glissando, etc.• ADSR envelope for entire sound

31 Oct. 07 22

Periodicity & Definite PitchPeriodic waveform: clearcut definite pitch

With sharp corners: much high-freq. energy

square wave; loud trumpet

Without sharp corners: little high-freq. energy

sine wave; low, soft flute

Almost-periodic waveform: definite pitch

piano

Somewhat periodic waveform: complex or multiple pitches

bells

Aperiodic waveform: noisy, no definite pitch

cymbal, bass drum

12 Feb. 07 23

Real-World Musical Sounds (1)• The “Attack/Sustain/Release” (ASR) model for notes

– Attack, Sustain, Release modified from recordings– Attack includes Decay from ADSR model

• Used in the Kurzweil 250 (1984), etc.– Original version had only 2 MB for all samples– Piano had diff. samples for 2 loudness levels– …and diff. sound for every 4-6 semitones– 1-2 sec. per sample for A+S+R

• How good did the K250 really sound?– COUNTDOWN, by Christopher Yavelow– “An opera for the nuclear age”

• “the ‘orchestral accompaniment’ is in reality a Kurzweil-250 digital sampler, synchronized to the baton of the conductor…”

– http://www.yavelow.com/docs/countdown.html

19 Feb. 07 24

Real-World Musical Sounds (2)

• Nowadays, can afford “unlimited” sustain• …but also need diff. sounds for many (8?) diff.

loudness levels (multisampling)– “All Together Now”, Electronic Musician, Jan. 2007

• …and diff. sound for every semitone or two• W/ unlimited sustain, takes gigabytes just for

piano!

29 Oct. 07 25

Timbre Space & Envelopes (1)• Timbre space

– Includes all possible timbres– Not well understood; may be 3-dimensional

• Original “FourierRandomTimbre” covers only a tiny part

• Some ways to enlarge it:– Overall envelopes– Allow (inharmonic) partials– Envelopes for partials

• Formants show timbre involves context: not really a property of a note!

• Timbre as "the psychoacoustician's multidimensional wastebasket category"

29 Oct. 07 26

Timbre Space & Envelopes (2)• Envelope

– Cf. Wikipedia article on timbre– Amplitude structure of a sound, so called because sound

just "fits" inside its envelope• Piecewise-linear envelope

– Envelope that can be drawn with a relatively small number of straight lines and no curves

– Important in music synthesis because can approximate many complex real-world sounds well with very little data

– Special case: "ADSR" (Attack - Decay - Sustain - Release) envelopes; common on synthesizers, etc.

– Samplers may combine A & D

22 Sep. 2006 27

Uncompressed Audio Files are Big

• 1 byte = 8 bits (nearly always)• How much data on a CD?

– CD audio is 44,100 samples/channel/sec. * 2 bytes/sample * 2 channels = 176,400 bytes/sec., or 10.5 MByte/min.

– CD can store up to 74 min. (or 80) of music– 10.5 MByte/min. * 74 min. = 777 MBytes– Actually more: also index, error correction

data, etc.

16 Feb. 06 28

Compressed Audio: Lossless & Lossy

• Don’t confuse data compression and dynamic-range compression (a.k.a. audio level compression, limiting)

• Codec = COmpressor/DECompressor• Lossless compression

– Standard methods (LZW: .zip, etc.) don’t do much for audio

– Audio specific methods

• MLP used for DVD-Audio• Apple & Microsoft Lossless

• Lossy compression– Depends on psychoacoustics (“perceptual coding”)

13 Feb. 06 29

Specs for Some Common Audio Formats

22 Sep. 2006 30

Psychoacoustics & Perceptual Coding

• Pohlmann, Ken (2005). Principles of Digital Audio, 5th ed., Chapter 10: Perceptual Coding

• Rationale: much better data compression• Physiology of ear and critical bands

– Not fixed frequency: any sound creates one or more critical bands

• Masking– Depends on relative loudness & frequency– Noise is much better than pitched sounds

• Threshhold of hearing– Depends greatly on frequency

22 Feb. 06 31

Compressed Audio: Lossy Compression

• General method1. Divide signal into sub-bands by frequency2. Take advantage of:

• Masking (“shadows”), via amplitude within critical bands

• Threshhold of audibility (varies w/ frequency)• Redundancy among channels

• MPEG-1 layers I thru III (MP-1, 2, 3), AAC get better & better compression via more & more complex techniques– “There is probably no limit to the complexity of

psychoacoustics.” --Pohlmann, 5th ed.– However, there probably is an “asymptotic” limit to

compression!• Implemented in hardware or software codecs

rev. 25 Sep. 2006 32

Another “Analog vs. Digital” Battle

• Merton, Orren (2006, February). The Sum of All Tracks. Electronic Musician 22,2, pp. 57-63

• About summing (mixing) with analog vs. digital hardware

• Tested with excellent hardware, panel of experts

• Methodology not really scientific• But is this fair? Does it really matter? How can

you do better with subjective phenomena?– Electronic Musician isn’t a scholarly journal– Critical: blind study => results probably meaningful– Cf. http://createdigitalmusic.com/2006/09/06/analog-

summing-pm8-for-people-who-dont-trust-software-mixing/

25 Sep. 2006 33

Scientific Evaluation of Subjective Phenomena: Lossy Compression (1)

• Pohlmann, Ken (2005). Principles of Digital Audio, 5th ed., pp. 403-413

• Discusses perceptual coding performance evaluation

• Want to evaluate as objectively as possible!• One idea: measure artifacts

– Difference between original & compressed => distortion

– Psychoacoustic model can estimate NMR (Noise-to-Masking Ratio)

25 Sep. 2006 34

Scientific Evaluation of Subjective Phenomena: Lossy Compression (2)

• Pohlmann: “best way to evaluate is to exhaustively listen”

• Tests with excellent hardware, panel of experts

• …but serious scientific methodology• Double-blind, ITU-R methods, CCIR 5-pt scale,

>=50 subjects, etc.• Similar methods used in validation of MP3,

AAC, etc.• …and ff123 et al’s evaluation of Ogg Vorbis• Our listening test: at http://www.informatics.

indiana.edu/donbyrd/N560Site-Fall06/Worst/

25 Sep. 2006 35

Auditory Illusions 1

• Wikipedia article “auditory illusion”• Deutsch's Musical Illusions

– E.g., Mysterious melody, Chromatic illusion– http://psy.ucsd

.edu/~ddeutsch/psychology/deutsch_research1.html

• Visual and Auditory Illusions (from UBC)– Often analogous illusions– E.g., Shepard's Tones, Tritone Paradox– http://www.cs.ubc.

ca/nest/imager/contributions/flinn/Illusions/Illusions.html

rev. 25 Jan. 2007 36

Auditory Illusions 2: Note, Chord, or Scale?

• Issue is partials each with its own envelope• Clue: relative frequencies of partials• Clue: shapes of envelopes• Try to simplify the sound…• Do you experience fusion?

rev. 5 Nov. 2007 37

Auditory Illusions: Shepard’s Tones

• Also called (less accurately) endless glissando• Demo on Web

– http://www.cs.ubc.ca/nest/imager/contributions/flinn/Illusions/Illusions.html

– Wikipedia version has a more interesting timbre

• Again, it’s partials & envelopes• Clue: relative freqs. of partials are important,

unusual• Clue: envelopes of partials are important• Used by J.C. Risset in computer-synthesized

music– in one case, with rhythm equivalent

rev. 25 Jan. 2007 38

Visual Analogue of Shepard’s Tones

rev. 5 Nov. 2007 39

Auditory Illusions: Mysterious Melody

• Hard to recognize, unless you already know what it is!

• Clue: probably easier for Americans