City Research Online · placement to reproduce the instrument’s geometry electroacoustically. Timbre Space In contrast to instrumental directivity patterns, one can give a different

City, University of London Institutional Repository

Citation: Einbond, A. (2017). Mapping the Klangdom Live: Cartographies for piano with two performers and electronics. Computer Music Journal, 41(1), pp. 61-75. doi: 10.1162/COMJ_a_00397

This is the published version of the paper.

This version of the publication may differ from the final published version.

Permanent repository link: http://openaccess.city.ac.uk/17394/

Link to published version: http://dx.doi.org/10.1162/COMJ_a_00397

Copyright and reuse: City Research Online aims to make research outputs of City, University of London available to a wider audience. Copyright and Moral Rights remain with the author(s) and/or copyright holders. URLs from City Research Online may be freely distributed and linked to.

City Research Online: http://openaccess.city.ac.uk/ [email protected]

City Research Online

Mapping the KlangdomLive: Cartographies forPiano with Two Performersand Electronics

Aaron EinbondDepartment of MusicCity, University of LondonNorthampton Square,London EC1V 0HB, [email protected]

Abstract: The use of high-density loudspeaker arrays (HDLAs) has recently experienced rapid growth in a wide variety oftechnical and aesthetic approaches. Still less explored, however, are applications to interactive music with live acousticinstruments. How can immersive spatialization accompany an instrument already with its own rich spatial diffusionpattern, like the grand piano, in the context of a score-based concert work? Potential models include treating thespatialized electronic sound in analogy to the diffusion pattern of the instrument, with spatial dimensions parametrizedas functions of timbral features. Another approach is to map the concert hall as a three-dimensional projection of theinstrument’s internal physical layout, a kind of virtual sonic microscope. Or, the diffusion of electronic spatial soundcan be treated as an independent polyphonic element, complementary to but not dependent upon the instrument’sown spatial characteristics. Cartographies (2014), for piano with two performers and electronics, explores each ofthese models individually and in combination, as well as their technical implementation with the Meyer SoundMatrix3 system of the Sudwestrundfunk Experimentalstudio in Freiburg, Germany, and the 43.4-channel Klangdomof the Institut fur Musik und Akustik at the Zentrum fur Kunst und Media in Karlsruhe, Germany. The process ofcomposing, producing, and performing the work raises intriguing questions, and invaluable hints, for the compositionand performance of live interactive works with HDLAs in the future.

Background

The richness and irreproducibility of acoustic instru-mental sound comes from complex interdependen-cies between timbre and space (Otondo et al. 2002).Only recently, however, do computer tools makeit possible for composers to strategize a high-levelcontrol of these dimensions for spatialized soundsynthesis, an ecriture of space and timbre (Warus-fel and Misdariis 2001). The recent confluence oftools for analysis and synthesis based on audiodescriptors, compositional control of spatialization,and high-density loudspeaker arrays (HDLAs) forrendering spatialized sound in three-dimensionsencourages synergies of timbre and spatial synthesisthat can be exploited for musical expression.

Instrument Directivity

Research into the spatial directivity patterns ofacoustic instruments goes back a half century;for a review see Frank Zotter’s PhD dissertation(2009, p. 89). Research in reproducing instrumental

Computer Music Journal, 41:1, pp. 61–75, Spring 2017doi:10.1162/COMJ a 00397c© 2017 Massachusetts Institute of Technology.

diffusion patterns is more recent, however. In theORA project (d’Alessandro et al. 2009), researchersat the Institut de Recherche et Coordination Acous-tique/Musique (IRCAM) explored the projection ofthe interior space of the pipe organ into the spacesurrounding the audience by applying techniquesfrom higher-order Ambisonics (HOA). In furthercase studies, instrumental directivity patterns weremeasured with a spherical microphone array andrecreated using Wave-Field Synthesis (WFS) andspherical loudspeaker arrays (Noisternig, Zotter,and Katz 2011). Another precedent is the use of acontrolled directivity source to reproduce instru-mental sound, such as the spherical 120-loudspeakerarray developed at the Center for New Music andAudio Technologies (CNMAT), Univeristy of Cali-fornia, Berkeley, in collaboration with Meyer Sound(cf. Avizienis et al. 2006). This allows for the de-tailed mapping of sound to a three-dimensionalfocused directivity pattern, in emulation of anacoustic instrument, implemented using WFStechniques (Schmeder and Noisternig 2010). A com-positional application of WFS and HOA combinedinteractively with a live instrument is Rama Got-tfried’s Flouresce for violoncello and electronics(Gottfried 2012). Although this project consideredcompositional analogies between instrumental tim-bre and space, it did not use spatial microphone

Einbond 61

placement to reproduce the instrument’s geometryelectroacoustically.

Timbre Space

In contrast to instrumental directivity patterns,one can give a different account of spatial andtimbral perception as a cognitive construct. Studiesby Grey (1977) and Wessel (1979) have pointed tothe multidimensional nature of timbral hearing,and to spatial models as possible representationsof this aural reasoning. Although questions remainopen about the details of timbral perception and itsvariation over musical and cultural contexts, timbrespace can nevertheless be taken as a fruitful modelto guide specific creative situations. Cartographiesembraces these dual relationships of timbre andspace, both as objective physical interaction andsubjective mental map.

Concatenative Synthesis

To retrieve fine timbral details of its sonic materials,Cartographies takes advantage of the technique ofcorpus-based concatenative synthesis, permittinghigh-level control of sound synthesis based on audiofeatures. Corpus-based concatenative synthesis hasbeen implement in the software CataRT, with itsassociated signal processing package FTM & Co., byDiemo Schwarz and the Sound Music MovementInteraction team at IRCAM (Schwarz 2007). CataRTallows for the analysis and segmentation of adatabase of samples recorded live or in deferredtime, the corpus, and resynthesis through to avariety of control paradigms. It gives access to thefull richness of time-domain audio, combined withthe fine control of sonic descriptors, defined as anycharacteristic extracted from the source sound orhigher-level features attributed to it.

Sounds are initially segmented algorithmically ormanually into short units, or grains. During synthe-sis, units are selected from the corpus based on theirdescriptors, usually according to Euclidean distancefrom desired target values. Units may be furthermanipulated using granular synthesis parametersbefore being overlapped or concatenated, and sent

to output. When the target descriptor values arederived from the analysis of another, longer targetsound, then the resulting synthesis can be describedas an audio mosaic. This mosaic may resemblethe target to varying degrees, as a visual mosaicmay resemble its subject. The user interface ofCataRT includes a multidimensional plot of unitsorganized by selected descriptors (see Figure 1).An analogy can be drawn between this representa-tion of timbral features and the notion of timbrespace.

Previous work extends corpus-based concate-native synthesis as a tool for computer-assistedcomposition and real-time treatment. In What theBlind See (2009), for five performers and electronics,each instrument is amplified with contact micro-phones (harp, piano, bass drum, and tam tam) orproximate microphones (viola and bass clarinet) toisolate its delicate inner sound world. These liveinstrumental signals are compared by a variety ofdescriptors to a prerecorded corpus of instrumentalsamples, also recorded by closely placed micro-phones. The samples are then concatenated duringperformance to produce a shadow of live instrumen-tal timbre (Einbond, Schwarz, and Bresson 2009).The same process can also be transcribed into in-strumental notation to be reinterpreted acoustically,a technique termed “corpus-based transcription.”In this way the timbre of a recorded sound—forexample a field recording—is mapped to a scorefor live performance. The resulting “instrumentalaudio mosaic” can fuse seamlessly with the sourcerecording; but the mosaic also can be used to maskor oppose its source, navigating different degrees ofrelation and reference.

What the Blind See also introduces the processof “corpus-based spatialization” to map timbraldescriptors to spatial trajectories (Einbond andSchwarz 2010). The goal is to guide real-timespatialization for recorded sounds, or even livesounds whose descriptor values are not known inadvance, along preplanned descriptor templates.Samples are compared to existing spatializationschemata according to their descriptor values andplaced in space at the appropriate location. As theinstrumentalists perform, the microscopic timbraldetails of their actions are analyzed and mapped

62 Computer Music Journal

Figure 1. Screenshot ofCataRT showing a samplecorpus (a database ofsamples) used inCartographies. Pointsrepresent grains of piano

samples that are plottedalong axes representingaudio descriptors, in thiscase MIDI note number(x-axis) and spectralcentroid (y-axis). The plot

is used to organize bothtimbral structure andspatialization strategies inthe composition.

to a spatial trajectory in the concert hall, whichmay differ from one performance to another. Likeacoustic instrumental directivity patterns, thispresents a correlation between timbral features andspatial distribution, albeit in a more abstract way.

Motivation for Cartographies

Cartographies extends corpus-based spatializationby projecting the inside of the piano out to immerse

the public in a magnified view of the instrument.A key dimension is the isolation of sonic detailsfrom their physical sources through recording andamplification. This could be compared to PierreSchaeffer’s “reduced listening,” or Helmut Lachen-mann’s “musique concrete instrumentale,” inwhich playing techniques approach independentsound objects with their proper forms and mor-phologies. The effect is enhanced by contact orproximity microphones placed in the piano inte-rior, where the performers act with their hands,

Einbond 63

Figure 2. Microphones andmaterials in the pianointerior at theExperimentalstudio.Materials used includealuminum foil, knittingneedles, Velcro, styrofoam,felt, and plastic wrap.

percussion mallets, and objects to elicit soundsbarely audible without amplification. The minutescale serves both to reduce the effect of masking ofamplified and spatialized sound by the live source,and to defamiliarize the sonic identity of this most“conventional” of instruments.

Unlike Lachenmann, for whom the physicality ofsonic gestures relates back to their “mechanical ori-gins” (Ryan 1999), the sound world of Cartographiesfocuses on nearly invisible actions that belie theirsources. Bent over the instrument, the performersproduce microscopic gestures that are obscuredfrom the public’s view; the results are transmittedthrough microphones and projected into the concertspace to reproduce a larger-than-life map of the pianointerior. See Figure 2 for a view of the installation ofthe microphones and some of the materials sampledand played by the performers inside the piano,including aluminum foil, knitting needles, Velcro,styrofoam, felt, and plastic wrap. Although someobjects resemble John Cage’s piano preparations inthe Sonatas and Interludes (1948), they differ inacting directly on the piano strings and frame, not

mediated by the keyboard and hammers. They aremore indebted to the inside-piano improvisationpractice of Andrea Neumann (Haenisch 2013).

Cartographies makes reference to an extramusicalsource as well: the cosmic microwave background(CMB), the faint electromagnetic radiation thatreaches the earth from distant space, possibleevidence of the first moments of the inflation of theuniverse after the Big Bang. Although this radiationsurrounds us in all directions, it only allows us dimlyto apprehend its source: like the sounds of the pianothat are only partially revealed through their captureand projection by a network of microphones anddome of loudspeakers. To underline this metaphor,a sonification of CMB data is incorporated as a“found sound object” at several points in thework.

Spatialization Models

During the composition and production of Cartogra-phies three spatialization models were explored

64 Computer Music Journal

and each incorporated at different points in thework: projection of the piano interior through ana-log amplification, corpus-based spatialization ofpiano samples, and independent spatial trajectoriesconvolved with piano resonances.

Projection

The first approach to spatialization is accomplishedin the simplest possible way: Twelve microphonesare installed in the grand piano at the Zentrum furKunst und Media (ZKM), and each is mapped to avirtual source position in the Klangdom. (The pianois a Yamaha seven-foot model S6, similar in dimen-sions to a Steinway B.) There are several differencesfrom other, earlier examples of three-dimensionalinstrumental recording and diffusion. First, suchrecording-diffusion scenarios are often applied indeferred time, whereas this was a real-time ap-plication. Second, although Ambisonic recordinginvolves regularly spaced microphone placement,here the microphones were positioned subjectively,based on the instrument’s geometry, tone color, andaccessibility for the performers (without obstructingtheir hand movements or sight lines). For this reasoncompact lavalier and transducer microphones wereused, as listed in Table 1. Last, although multi-channel microphone placement usually attempts tocapture the instrument’s diffusion pattern throughair, for this project microphones were positioned ei-ther in extreme proximity to or in contact with theirsound sources. Rather than capturing a “realistic”sonic image of the piano to be reproduced virtually,this approach favored a more abstract mapping thatcould be manipulated artificially.

The chosen microphone positions and virtualsource positions are shown in Figures 3 and 4.Initial experimentation revealed that, when theinternal positions of the microphones are mappedconsistently to diffusion positions, recognizablespatial effects can be reproduced. For example,when “circular bowing” with a contrabass bow collegno—using the wood of the bow to design largecircular arcs left and right across the metal stressbar of the instrument between transducers T2, T3,and T4—a corresponding left–right motion is heard

Table 1. Microphones and OutputRouting

Microphone Position Output

Schaller T1 (mobile) L1-6DYN-P T2 L6DYN-H T3 L7DYN-P T4 L2DYN-P T5 L5DYN-P T6 L4DPA4099 M7 L1DPA4060 M8 L2DPA4099 M9 L3ME104 M10 L6ME104 M11 L5BLM03C M12 (floor or lid) L8

The microphones used were: Schaller Oysterpiezo; Schertler DYN-P and DYN-Htransducers; DPA 4060, 4099, and SennheiserME104 lavalier microphones; and SchoepsBLM03C boundary microphone.Refer to Figures 3 and 4 for microphone layoutand output positions.

from the corresponding source positions, L6, L7,and L2.

This approach to sound projection takes advan-tages of the unique collaboration between ZKM andthe Sudwestrundfunk Experimentalstudio (EXP),and the particular technological and human capa-bilities available through this collaboration. TheExperimentalstudio has developed a unique ap-proach to sound diffusion, eschewing regularlyspaced circular loudspeaker arrays in favor of tailor-made loudspeaker arrangements for each project.An emblematic example is Luigi Nono’s . . . sofferteonde serene . . . where stereo loudspeakers are com-plemented by a third speaker under the instrumentfor maximal fusion with piano samples in the tapepart. In Cartographies, even though the final dif-fusion system was the symmetrical HDLA of theKlangdom, the virtual source positions were treatedmore subjectively and could be adjusted duringrehearsals based on listening criteria.

The approach is enhanced by the electronic perfor-mance practice of EXP, with three computer musicengineers each following a score of the work. Oneengineer is responsible for advancing the electronic

Einbond 65

Figure 3. Microphonepositions inside theYamaha S6 piano used inCartographies.

Figure 3

Figure 4. Positions ofvirtual loudspeakers,piano, and mixing console.The virtual loudspeakerscorrespond to adjustablepositions in the Klangdom,interpolated using

Zirkonium software. SeeTable 1 for the mapping ofthe piano microphonepositions to the positionsof the virtualloudspeakers.

Figure 4

cues, one for the input and output levels of the liveprocessing, and one for the input and output levelsof the instrumental amplification. This permits alevel of electroacoustic craft not easily achievedotherwise. In the case of Cartographies, the scoreindicates changes in the levels of different micro-

phones at specific moments based on their locationsand timbres. For example, near the end of the work,the levels of the five Schertler transducers affixed tothe piano frame are raised while other microphonesare silenced, producing a specific color filtered bythe metal frame and limited frequency response ofthe transducers. As the microphones are mappedto fixed virtual locations in the Klangdom, theseperformance instructions also affect the spatializa-tion pattern, creating a dynamically shifting mapthroughout the performance.

Corpus-Based Spatialization

A contrasting approach is corpus-based spatializa-tion, where descriptor values analyzed from the liveperformers are used to pilot spatial trajectories ofprerecorded samples. This is promising for its poten-tial to create spatial motions responding to timbraldescriptors analyzed dynamically in performancewithout relying on preprogramed trajectories. Differ-ent models for the synthesis of spatial motion wereexplored: In the simplest case, grains are concate-nated in a monophonic audio channel that is thenmoved to a new virtual source location dependingon the most recent grain synthesized. Althoughthis leads to a clear perception of spatial motion, itproduces a jerky effect by suddenly displacing grainsas they are sounding. Alternately, each grain istreated as a separate polyphonic voice, remaining ata constant spatial location for its duration, regardlessof the location of subsequent grains. This fills thespace more vividly, yet still allows for the percep-tion of virtual motion between grains. An acousticanalogy could be to a percussionist surrounded bya large collection of small instruments: Althougheach instrument remains stationary, we experiencethe performer’s spatial gesture through the sequenceof interactions with the instruments.

In Cartographies, the performers trigger corpus-based spatialization through a process of live audio“mosaicking.” As the performers interact with thepiano, their sound is captured through differentcombinations of microphones whose input levels tothe software are regulated by the audio engineers,once again permitting fine control over microphone

66 Computer Music Journal

position and color. This input signal is used totrigger audio mosaicking by CataRT based on theprerecorded database of piano samples. The corpussamples were recorded using the same microphonesetup as in the performance to ensure that theyare as similar as possible. The same sample databasewas also used to compose the instrumental scoreof the work through the process of corpus-basedtranscription: Target sounds, drawn from longerpiano samples as well as the found sound of theCMB, were transcribed into music notation readableby the performers, producing an audio mosaic tobe reinterpreted instrumentally. When the sourcesound, for example Performer 1 rubbing a piezomicrophone against a piece of aluminum foil,is reinterpreted in performance along with itsmosaicked transcription by Performer 2, the resultis a close timbral fusion between the performers. Adetailed account of the transcription process usingCataRT and the Bach package for Max is given in anearlier paper (Einbond et al. 2014).

Independent Trajectories

Finally, a third spatialization model was explored inwhich the spatial trajectory of the electronic soundwas composed in counterpoint to the acoustic piano.This could be thought of as the default configurationfor spatialization in music for instruments and elec-tronics. As an example, Cartographies begins withan analog input signal panned in a circular motionaround the outer ring of virtual output positions(L1–6), realized with two Halophones rotating at dif-ferent speeds. (This technique, named after hardwarebuilt by longtime director of EXP Hans Peter Haller,is now implemented with a Max patch.) Cartogra-phies takes this idea further, however. The MeyerSound Matrix3 system incorporates the softwaretool SpaceMap (Ellison 2013), permitting a high levelof customization and control of spatial trajectories.A spatial path can be recorded in real time usingthe mouse, then further edited in deferred time, andplayed back on cue in performance (see Figure 5).At the suggestion of Reinhold Braig (sound directorand music computer engineer at EXP responsiblefor the production of Cartographies), this tool was

used not only to create virtual spatial trajectories,but also to pilot signal processing by routing theoutputs of SpaceMap to other real-time treatments.As an input channel is moved along a trajectory,its relative output levels to different processes aregradually cross-faded, and their outputs may, inturn, be positioned spatially.

This technique was used to apply differentshadings of an impulse-response (IR) convolutionreverberator to live input. The idea was to fusethe input sound with the resonance of the pianoby convolving it with IR models sampled fromimpulses on the metal frame. Impacts of differentpercussion mallets (snare drum stick, yarn mallet,and bass drum beater) were recorded using four of theSchertler transducers mounted on the frame (T2, T3,T4, and T6 in Figure 3). Sounds convolved with theseIR samples took on some of the coloring of the metalframe and the Schertler microphones themselves,similar to the sounds amplified using the same mi-crophones. The convolvezerolatency.maxpatabstraction, from the AHarker library, was used forreal-time processing, as it offers an ideal balance ofsound quality, low latency, and manageable CPUcost. Please see the HISSTools Impulse ResponseToolbox (cf. Harker and Tremblay 2012) for theupdated external multiconvolve∼.

The outputs of the four IR treatments werethen routed to virtual source positions L6, L3, L4,and L5. In addition to the four convoluted signals,virtual source positions L1 and L2 were reservedfor the untreated input signal, with levels carefullybalanced by ear with the quieter, treated outputs.Trajectories were recorded and edited in SpaceMapto route the input signal in a circular motion to thesesix output channels, four treated and two untreated,producing a gradual cross-fade of untreated andconvoluted signal accompanying the spatial motion.

For the live input here, as well as at the work’sopening, the Schaller Oyster piezo microphone (T1)was selected as Performer 2 used it to interact withvarious materials (aluminum foil, plastic wrap,felt, scrub brush, and sponge) as well as the pianointerior. Because of the coloring of the piezo andthese “foreign” objects, this was one of the least“pianistic” sounds in the work. By convolving itwith the IR samples, however, it was brought into

Einbond 67

Figure 5. Screenshot ofMeyer Sound’s SpaceMap.Squares at the cornersrepresent virtualloudspeaker positions, andthe curve represents a

hand-drawn spatialtrajectory that can beplayed back to simulatethe motion of virtualsources.

a closer fusion with the sound world of amplifiedpiano.

Implementation

The concert patch for Cartographies builds on previ-ous approaches to live treatment and spatialization(Einbond, Schwarz, and Bresson 2009; Einbond andSchwarz 2010), with novel features such as the useof a custom, expanded version of CataRT, commu-nication between Max and Zirkonium software, and

communication between the Matrix3 system of EXPand the ZKM Klangdom.

Matrix

The Experimentalstudio makes use of MeyerSound’s Matrix3 Audio Show Control System(www.meyersound.com/products/matrix3), com-bining low-latency hardware inputs and outputswith Open Sound Control (OSC) communication.It is controlled by custom-built fader surfaces that

68 Computer Music Journal

allow access to faders for physical input and outputchannels, as well as programmable “virtual faders”to send OSC messages to Max. Owing to the prac-tical limitations of the collaboration between EXPand ZKM, a single piece of Meyer LX-300 hardwarewas used, limiting the input to the Klangdom to 16channels. To get the most out of the 43.4 channelsof the Klangdom (the configuration in 2014), these16 channels were mapped to 16 virtual sourcesthat could be positioned dynamically using ZKM’sZirkonium software (Brummer et al. 2014). The16 channels of audio were sent over a Dante digi-tal audio network, and spatialization informationwas sent as OSC messages from Max to Zirko-nium over a local network using the Max objectsudpsend and udpreceive. Eight channels weremapped to fixed virtual source locations withinthe Klangdom, to be used for live amplification(see Figures 3 and 4) and some live processing,and the other eight channels were mapped to dy-namically moving virtual sources, to be used forcorpus-based spatialization with CataRT. Ratherthan controlling the levels of the eight shiftingsources individually, their master levels were in-stead controlled by two “virtual faders” sent fromthe control surface back to the concert patch usingOSC messages. Other treatments, such as IR con-volution, were mapped to the fixed eight outputpositions.

CataRT

Cartographies adds to previous musical work withCataRT by using a custom version with a modulardescriptor analysis framework as described bySchwarz and Schnell (2010). It includes an extendeddescriptor list based on the ircamdescriptors∼analysis module, allowing for more fine control ofa large selection of timbral and other features, asdefined by Peeters (2004).

Choice of Corpora and Descriptors

Two corpora of piano samples were chosen for usein the concert patch: Corpus 1, recorded with theSchaller Oyster piezo microphone, and Corpus 2,

recorded with the Schoeps BLM03C boundary micro-phone. The former provided a filtered, compressed,“electronic-sounding” version of the piano, and thelatter gave a broadband, “naturalistic” sound. Theinput from any combination of microphones couldbe routed to either or both corpora to trigger liveaudio mosaicking. This provides a wide range ofcoloristic choices, as well as the possibility to assigntreatments to only one of the two performers byselecting the microphones closest to the performer’srange of action inside the piano. For example, sum-ming the signals from M7, M10, and M11 triggerssynthesis primarily in response to Performer 1,whereas M8 and M9 respond primarily to Performer2 (for reference see Figure 3).

For corpus-based spatialization, two descriptoraxes were chosen, taking into account the distri-bution of the grains in the corpus, the perceptualsalience of the timbral axes, and the fusion of thistimbral distribution with the amplified piano sound.After testing several possibilities, the same pairof axes was used for both sample corpora: pitchalong the x-axis (specifically, the MIDI note numberanalyzed by the Yin algorithm; cf. de Cheveigneand Kawahara 2002), and brightness (spectral cen-troid) along the y-axis. These axes correspond wellto descriptor variance within the sample corpora,which contains both pitched sounds (such as highpiano strings strummed with wire brushes) andband-limited noise (such as low strings, preparedwith aluminum foil and knitting needles, and hitwith the pianist’s palm). These axes are not neces-sarily the same as the axes chosen for live-audiomosaicking, for which more than two descriptorscould be used to select timbres with greater preci-sion, and could be changed to suit the audio contentat different points in the work. In addition to pitchand spectral centroid, descriptors include spectralspread (a measure of bandwidth), level (in dB), andperiodicity (a measure of noise content output bythe Yin algorithm).

Spatial Mapping

The spatialization axes were oriented with lowvalues toward the front right of the room and highvalues toward the rear left. This choice, along with

Einbond 69

the approximately triangular distribution of thegrains (see Figure 1), suggests a loose analogy tothe shape of the spatialized piano itself: its lowstrings projected across the front of the room andits high strings projected to the rear left (compareto Figures 3 and 4). The decision to use a singlemapping of descriptor space for the entire workechoes the fixed projection of the amplified piano.In both cases the listener is surrounded by a stable,coherent virtual timbre space that is slowly revealedover the course of listening.

The concert patch allows for the range of spatialx- and y-positions to be scaled to minimum andmaximum descriptor values, to the mean ± standarddeviation, or to an arbitrary interval. Initial listeningtrials revealed that mean ± standard deviation gavethe best results, mapping the most dense part of thecorpus to a central location. Source distance is nottaken into account by the spatialization algorithmused in the Zirkonium software, so only two spatialdegrees of freedom are left: azimuth and elevation.Therefore, to map the two-dimensional descriptorspace from CataRT to the three-dimensional Klang-dom, the z-position of each grain was projected tothe surface of the unit sphere x2 + y2 + z2 = 1for x2 + y2 < 1, and 0 elsewhere, as given byz =

√max (1 − x2 − y2, 0).

Virtual Polyphony

Under other circumstances, CataRT could be usedfor spatial positioning in an arbitrary number ofchannels: The module catart.synthesis.multicommunicates with the external object gbr.ola∼from library FTM & Co. to control an efficientoverlap-add algorithm. For this production, how-ever, because only eight mobile output channelswere available to send to Zirkonium, a differ-ent approach was used to take advantage of thefull 43.4 channels of the Klangdom. The objectscatart.synthesis.multi and gbr.ola∼ are stillused, but each grain is mapped to a separate channelwith an amplitude of one, while spatial informationfor that grain is simultaneously sent to Zirkonium.A “busy map” is generated at the output of theconcert patch, returning a list of available channelsto catart.synthesis.multi before it synthesizes

the next grain. Conceptually similar to a poly∼object in “voice-stealing” mode, it limits synthesisto only eight simultaneous virtual spatial sources.Because the grains used in the piece are typicallyshort (250–1000 msec), however, this limitation wasnot perceived to be significant and was outweighedby the benefit of using the entire Klangdom. Ac-cording to the voice-stealing algorithm, if all eightchannels are already occupied, the next grain isoverlapped with the grain that has been playing thelongest—the entry effectively masking the motionof the previously sounding grain.

Following EXP practice, the live electronictreatments other than CataRT are sent as separatestems to the Matrix3 system, so that each levelcan be controlled manually before being routedto the eight fixed output channels in Zirkonium.Eight premix outputs from the concert patch areneeded: four IR convolutions, a bandpass filter usingbiquad∼, distortion using overdrive∼, and twochannels of sound file playback (serving for bothmonophonic and stereo files). See Figure 6 for ascreenshot of the concert patch.

Zirkonium

The Zirkonium software was developed at ZKM asa free OSX-based spatialization interface (Brummeret al. 2014). It uses the vector base amplitudepanning (VBAP) algorithm (Pulkki 1997) to mapvirtual source positions to triples of loudspeakers.Zirkonium responds to OSC messages specifyingpositions, allowing for flexible and legible control.In Cartographies, the 16 channels received from theMatrix3 system are mapped to 16 input channels inZirkonium. To limit possible rapid position changes,all OSC messages to Zirkonium are filtered with aspeedlim object set to 20 milliseconds, balancingsmooth spatial motion and CPU cost.

The eight fixed virtual source positions (channels1–8) are stored in a text file loaded into a collobject for easy editing. This allows them to bepositioned by ear during rehearsal in the concertspace. Rather than relying on theoretical positions,the particular spatial and acoustic characteristics ofthe room are taken into account in deciding how to

70 Computer Music Journal

Figure 6. Screenshot ofCartographies concertpatch showing audioinputs (upper left) andoutputs (lower left) forreal-time treatments.

Spatialization controlssent via udpsend to theZirkonium software arevisualized using theambimonitor objects onthe lower right.

project the amplified piano in an immersive way.On initialization of the concert patch, these eightpositions are recalled and sent to Zirkonium andchannels 9–16 are positioned at the origin, servingas visual confirmation that OSC communicationbetween Max and Zirkonium is functioning prop-erly. Figure 7 shows the final positions chosen forchannels 1–8 as well as the default positions forchannels 9–16.

Discussion

As a work for live instruments and interactiveelectronics projected over an HDLA, Cartographiespresents an uncommon scenario, and the particu-larities of the collaboration between EXP and ZKM

played a large part in the work’s successful realiza-tion. Yet given these special conditions, it is worthconsidering how this or a similar work could becarried forward to other performance situations orrealized in a scaled-down version.

[Editor’s note: a binaural mix with video ofCartographies is available at https://youtu.be/XbmGgYoEezU]

Listening Observations

Although certainly subjective, and localized to asingle performance realization, the author’s listeningobservations at the premiere may nonethelessbe instructive for future implementations andextensions of corpus-based spatialization. The

Einbond 71

Figure 7. Concert patch(details) showing initialpositions of outputs 1–16listed as rows of a colland displayed with theambimonitor object (a).(The ambimonitor detail

as seen on the lower rightof Figure 6.) Screenshot ofZirkonium showing sourcepositions (pentagons)superposed over the 43speakers (squares) of theZKM Klangdom (b).

choice of descriptor axes of pitch and brightnessfor spatialization worked well to spread the corpussounds across the Klangdom, as did descriptorscaling to the range defined by mean ± standarddeviation, so that the center of weight of thedistribution was shifted to the center of the room.Still, as visible in Figure 1, more samples weregathered toward the front of the room, coincidingwith the location of the stage. Yet this densityhad the positive effect of concentrating additionallistening focus in the direction of the piano andperformers. The importance of the subjectivechoice of this mapping should, therefore, not

be underestimated. A possible future directionwould be to use principal component analysis toidentify the descriptors, or linear combinations ofdescriptors, that produce the most variance acrossthe corpus, and therefore best distribute units acrossthe space. Such a calculation must still be evaluatedsubjectively through listening, however.

A related observation is that corpus-based spa-tialization produces a productive reinforcementbetween audio parameters. When, for example, agranular gesture from the right to the left of theroom is accompanied by a rise in brightness, thetrajectories reinforce each other, making the effectmore robust perceptually. This redundancy helps toensure that spatial and timbral effects are salientfor listeners positioned at different positions in theroom, not only those at the “sweet spot.” Thismitigates one possible drawback of HDLAs: Duringthe rehearsals and performance, listening positionsat many different locations in the Klangdom werefound to yield satisfactory timbral and spatial auralexperiences.

The “voice-stealing” algorithm, limiting syn-thesis to eight channels of virtual polyphony, wasrelatively successful in communicating an immer-sive granular texture without audible artifacts.Monitoring the patch during performance revealedthat some voice-stealing was taking place, in thesense of grains being moved to new locations whilestill sounding (not in the sense of grains beinginterrupted). These displacements were not readilyaudible, however, likely due both to the fast decay ofthe relatively short grains and to the psychoacousticprecedence effect. Owing to technical limitations,more than eight channels of virtual polyphonywere not tested in the Klangdom. This restriction,necessitated by communication between the EXPand ZKM systems, would be lifted, however, if thework were realized using a single system (discussedsubsequently in the section on “Portability Beyondthe Klangdom”).

Live Interpretation and Spatialization

The unique affordances of EXP performance practice,with its attention to live diffusion of both amplified

72 Computer Music Journal

and synthesized sound, became an important partof the work’s identity. The two instrumental per-formers and three electronic performers must adjusttheir interpretation in response to each other and theacoustic conditions of the space. The details of thescore and concert patch, although leaving enoughlatitude for inter-performance variation, also en-sures enough predictability for effective rehearsal.The goal is a “chamber music–like” interactionbetween the instrumentalists and engineers. If thework were realized by fewer engineers, especially ifthey were brought in at a late stage of the rehearsalprocess, the same level of control would be difficultto master. Also, programming more level changesinto the patch would have the disadvantage of addinglatency, as well as reducing sensitivity to timing andlevel variations by the live instrumentalists. Nev-ertheless, given the possible practical constraints offuture performances, alternative solutions must beconsidered.

Portability Beyond the Klangdom

By realizing Cartographies with Zirkonium soft-ware, a degree of portability is already assured:Zirkonium’s VBAP algorithm is independent ofspecific loudspeaker setup. The first step afteropening the Zirkonium Spatialization Server is toload a predefined speaker configuration as an XMLfile. Separate files describe the speaker positionsof ZKM’s Klangdom, the 24-channel “minidome”(where some of Cartographies was prototyped),or other systems. In theory, the production couldbe taken to any HDLA, the appropriate speakerconfiguration could be loaded, and the spatialsetup would be transparently adapted to the newsystem.

A greater degree of portability is needed to realizerehearsal and performance versions of the work witha reduced loudspeaker setup, however. The distribu-tion of eight virtual sources (shown in Figure 4) wasconceived with this possibility in mind, to be re-placeable by eight physical loudspeakers in the samepositions. This was the setup used for rehearsals atEXP before traveling to ZKM. The amplification and

live treatments (IR, filtration, distortion, sound-fileplayback) are easily adapted: Instead of sending eightchannels from the Matrix3 to Zirkonium, they aresent directly to the loudspeakers. Channels 1–6 arepositioned in a ring around the space, and channels7 and 8 are positioned near the center, raised, andpointed upward to simulate the upper speakers ofthe Klangdom.

The only part of the concert patch that requiresadjustment is concatenative synthesis with CataRT,which is now used for spatialization over the sameeight physical outputs as for the other treatmentsand amplification. This is easily achieved with thebuilt-in catart.synthesis.multi and gbr.ola∼modules, returned to their usual function to performspatialization with an overlap-add algorithm. Anonboard VBAP object is used to calculate spatialcoefficients based on the eight loudspeaker po-sitions, sent to catart.synthesis.multi andgbr.ola∼ before each grain is synthesized throughtthe eight outlets. The eight amplitude coefficientsfrom VBAP are applied independently to each grain,so an arbitrary number of virtual spatial positionscan be superposed. Unlike the Zirkonium versionof the patch, now corpus-based spatialization isrealized with unlimited virtual spatial polyphony,an advantage despite the smaller loudspeaker setup.

Finally, a still “lighter” version of the workis planned for performances at which EXP is notpresent, dispensing with the Matrix3 system infavor of a single Max concert patch. In this case,the built-in CataRT and VBAP modules will beused for spatialization as described earlier. Theremaining treatments, rather than being sent toMatrix3, will be routed directly in the patch usinga second VBAP module to replace SpaceMap. Thiswill have the disadvantage that live electronictreatments and amplification can no longer becontrolled independently from the mixer. But, as acompromise solution, a MIDI controller such as theBehringer BCF2000 can be added to adjust premixlevels in the patch during performance. Althoughthis light version is imagined for the eight-channelsetup described herein, thanks to the built-in VBAPmodules, it could be transparently scaled to largeror smaller loudspeaker setups.

Einbond 73

Future Directions

CataRT-style concatenative synthesis has now beenimplemented in the MuBu package for Max (Schnellet al. 2009), including many of the capabilities ofthe custom version used in Cartographies. Ratherthan catart.synthesis.multi, spatialization isperformed by mubu.concat∼, which can be set togenerate an arbitrary number of channels, to whichamplitude coefficients derived from VBAP can beapplied for corpus-based spatialization. A modularanalysis framework permits the use of the full rangeof ircamdescriptors∼ for fine timbral control.Based on Max externals rather than FTM, MuBubenefits from greater stability and portability aswell as access to 64-bit computing to permit largerdatabase sizes.

Recent research combines MuBu with a computerimprovisation algorithm based on the PyOracle li-brary for Python (Surges and Dubnov 2013). Theresulting tool, CatOracle (part of Forum IRCAM’sMuBu package: http://forumnet.ircam.fr/product/mubu-en/), can concatenate novel sequencesof grains based on shared context in descriptorspace (Einbond et al. 2016). Combining it withcorpus-based spatialization would have powerfulconsequences as a tool for spatialized improvisationand composition, complementing spatial logic withlearning and generation of musical structure. A firstapplication, Xylography for cello and electronics(2015), experiments with projecting the signal ofthe amplified cello, captured by four contrastingmicrophones (DPA 4060 miniature omni micro-phone, AKG C411 piezo microphone, and twoSchertler DYN-P transducers), across a four-channelloudspeaker system.

A longer-term goal is to incorporate research ininstrumental directivity patterns more directly intocorpus-based spatialization. Rather than focusing onthe physical geometry of the instrument or a simpletimbral-spatial mapping, as in Cartographies, thiswould imply directly modeling the directivity pat-tern projected from the instrument into the concertspace. By measuring, in a controlled environment,different timbral descriptors as functions of space,these distributions could be used as templates formapping concatenated grains to spatial locations ac-

cording to their descriptor values. In contrast to thelinear mapping used in Cartographies, this wouldpermit a more fine-grained distribution of timbre inspace, and promise an even closer fusion of acousticinstrumental directivity with live corpus-basedspatialization.

Acknowledgments

I thank Reinhold Braig and Gary Berger, sounddirectors and music computer engineers responsi-ble for the production of Cartographies; DominikKleinknecht and Simon Spillner, music computerengineers for the premiere of Cartographies; DetlefHeusinger and the team of Sudwestrundfunk Exper-imentalstudio; Gotz Dipper, David Wagner, LudgerBrummer, and the team of the Institut fur Musikund Akustik at ZKM, Karlsruhe; Diemo Schwarzand the Sound Music Movement Interaction teamat IRCAM; and Rei Nakamura and Olaf Tzschoppe,collaborators and performers of the premiere ofCartographies with the ZKM Klangdom on 25November 2014.

References

Avizienis, R., et al. 2006. “A Compact 120 Independent El-ement Spherical Loudspeaker Array with ProgrammableRadiation Patterns.” In Proceedings of the 120th Con-vention of the Audio Engineering Society. Availableonline at www.aes.org/e-lib/browse.cfm?elib=13587(subscription required). Accessed November 2016.

Brummer, L., et al. 2014. “New Developments forSpatial Music in the Context of the ZKM Klang-dom: A Review of Technologies and Recent Pro-ductions.” Divergence Press 3. Available online atdx.doi.org/10.5920/divp.2015.36. Accessed November2016.

d’Alessandro, C., et al. 2009. “The ORA Project: Audio-Visual Live Electronics and the Pipe Organ.” InProceedings of the International Computer MusicConference, pp. 477–480.

de Cheveigne, A., and H. Kawahara. 2002. “YIN, aFundamental Frequency Estimator for Speech andMusic.” Journal of the Acoustical Society of America111(4):1917–1930.

74 Computer Music Journal

Einbond, A., and D. Schwarz. 2010. “Spatializing Timbrewith Corpus-Based Concatenative Synthesis.” InProceedings of the International Computer MusicConference, pp. 72–75.

Einbond, A., D. Schwarz, and J. Bresson. 2009. “Corpus-Based Transcription as an Approach to the Compo-sitional Control of Timbre.” In Proceedings of theInternational Computer Music Conference, pp. 223–226.

Einbond, A., et al. 2014. “Fine-Tuned Control of Concate-native Synthesis with CataRT Using the Bach Libraryfor Max.” In Proceedings of the International ComputerMusic Conference, pp. 1037–1042.

Einbond, A., et al. 2016. “Introducing CatOracle: Corpus-Based Concatenative Improvisation with the AudioOracle Algorithm.” In Proceedings of the Inter-national Computer Music Conference, pp. 140–146.

Ellison, S. 2013. “SpaceMap: 20 Years of Audio Origami.”Lighting and Sound America (April):80–88.

Gottfried, R. 2012. “Studies on the Compositional Use ofSpace.” Technical report. Paris: Institut de Rechercheet Coordination Acoustique/Musique. Available on-line at articles.ircam.fr/textes/Gottfried13a/index.pdf.Accessed November 2016.

Grey, J. M. 1977. “Multidimensional Perceptual Scalingof Musical Timbres.” Journal of the Acoustical Societyof America 61(5):1270–1277.

Haenisch, M. 2013. “Materiality and Agency in Im-provisation: Andrea Neumann’s ‘Inside Piano’.” InA. Cassidy and A. Einbond, eds. Noise in and as Music.Huddersfield, UK: University of Huddersfield Press,pp. 147–170.

Harker, A., and P. A. Tremblay. 2012. “The HISSTools Im-pulse Response Toolbox: Convolution for the Masses.”In Proceedings of the International Computer MusicConference, pp. 148–155.

Noisternig, M., F. Zotter, and B. F. G. Katz. 2011.“Reconstructing Sound Source Directivity in VirtualAcoustic Environments.” In Y. Suzuki, D. Brungart,and H. Kato, eds. Principles and Applications ofSpatial Hearing. Singapore: World Scientific, pp. 357–373.

Otondo, F., et al. 2002. “Directivity of Musical Instru-ments in a Real Performance Situation.” In Proceedingsof the International Symposium on Musical Acoustics,pp. 312–318.

Peeters, G. 2004. “A Large Set of Audio Features forSound Description (Similarity and Classification) in theCUIDADO Project.” Technical report. Paris: Institutde Recherche et Coordination Acoustique/Musique.Available online at recherche.ircam.fr/anasyn/peeters/ARTICLES/Peeters 2003 cuidadoaudiofeatures.pdf.Accessed November 2016.

Pulkki, V. 1997. “Virtual Sound Source Positioning UsingVector Base Amplitude Panning.” Journal of the AudioEngineering Society 46(6):456–466.

Ryan, D. 1999. “Composer in Interview: Helmut Lachen-mann.” Tempo (210):20–24.

Schmeder, A., and M. Noisternig. 2010. “SpatializationSchemata for High-Order Source Directivity.” Paperpresented at the GDIF/SpatDIF Meeting, Institutde Recherche et Coordination Acoustique/Musique,Paris.

Schnell, N., et al. 2009. “MuBu and Friends: Assem-bling Tools for Content Based Real-Time InteractiveAudio Processing in Max/MSP.” In Proceedings ofthe International Computer Music Conference, pp.423–426.

Schwarz, D. 2007. “Corpus-Based Concatenative Synthe-sis.” IEEE Signal Processing Magazine 24(2):92–104.

Schwarz, D., and N. Schnell. 2010. “A Modular SoundDescriptor Analysis Framework for Relaxed-Real-TimeApplications.” In Proceedings of the InternationalComputer Music Conference, pp. 76–79.

Surges, G., and S. Dubnov. 2013. “Feature Selection andComposition Using PyOracle.” Paper presented at theAIIDE Workshop on Musical Metacreation, October 14–15, Northeastern University, Boston, Massachusetts.Available online at www.aaai.org/ocs/index.php/AIIDE/AIIDE13/paper/view/7452/7685. AccessedNovember 2016.

Warusfel, O., and N. Misdariis. 2001. “Directivity Synthe-sis with a 3D Array of Loudspeakers, Application forStage Performance.” In Proceedings of the InternationalConference on Digital Audio Effects. Available online atwww.csis.ul.ie/dafx01/proceedings/papers/warusfel.pdf.Accessed November 2016.

Wessel, D. 1979. “Timbre Space as a Musical ControlStructure.” Computer Music Journal 3(2):45–52.

Zotter, F. 2009. “Analysis and Synthesis of Sound-Radiation with Spherical Arrays.” PhD dissertation,Institute of Electronic Music and Acoustics, Universityof Music and Performing Arts, Graz.

Einbond 75