Applications of computers to dance

Of all the art forms, dance has probably been theslowest to adopt technology. In part, this reflects

the reluctance of dancers and choreographers to letanything get between them and the live kinestheticexperience. But it also reflects the fact that useful toolshave been slow to develop in a market with limited com-mercial opportunity.

Probably the first article suggesting the applicationof computers to dance was published by A. Michael Nollin Dance Magazine in January 1967, although New Yorkchoreographer Merce Cunningham discussed this atabout the same time.1 These initial discussions sug-gested visualizing dance with animated stick figures andusing these visualizations to plan choreography. Soonother choreographers suggested using computers tocompose and edit dance notation scores and then totranslate these scores into animation. Dance notation isanalogous to music notation, but is less widely known—the “Labanotation Primer” sidebar provides a primer onthe notation system most widely used in North America.

As computer technology has developed and becomeless expensive, many artists have found ways to use it toenhance their performances with interactive multime-dia. This has included incorporating computer-generatedimages and sound with live dance performance andusing sensors that let the live dancers’ movements con-trol imagery, sound, and a wide variety of special effects.

This overview describes some of the current applica-tions of computer graphics to dance including visualiz-ing choreography, composing, editing and animatingdance notation, and enhancing live performance.

Visualizing dance and planningchoreography

When composing a dance, a choreographer typicallyworks with one or more dancers to physically set themovement of each dancer over time. For any significantpiece, this can take weeks or months. The choreogra-phy requires not only the dancers’ time but also accessto appropriate rehearsal space in a dance studio. This isexpensive and dance companies work on restricted bud-gets. A computer tool that could be used to plan thechoreography would help minimize the time needed forlive rehearsal.

In 1991, the founders of Credo Interactive—the devel-oper of Life Forms, a software package for planning

human movement—began to work with Cunninghamand other choreographers to explore this approach. Theresult was the evolution of Life Forms into a tool knownas Life Forms Dance and now as DanceForms(http://www.danceforms.com). This tool lets the chore-ographer try out ideas before ever meeting with livedancers. None of DanceForms’ features would surprisethose working in human figure animation; however, it’scustomized for dance. Similar features exist in CuriousLabs’ Poser, Alias’ Motion Builder and in general-purposeanimation systems such as Alias’ Maya and Discreet’s 3dsmax. However, most of those applications are opaque tothe typical dance choreographer or dance teacher.

As Figure 1 (page 8) shows, DanceForms provides thechoreographer with a stage, where figures can beplaced. While it’s possible to use any body with an artic-ulated skeleton (for example, a spider, a horse, and soon), the standard bodies provided are male and femalehuman figures for ballet and modern dance. After select-ing a figure or figures, a keyframe animation approachbuilds up the movement for each of the dancers. A stu-dio window provides a flexible set of controls to refinethe posture of individual figures, including 3D poten-tiometers for joint angles. Limbs and limb segments canbe directly manipulated with the mouse, with or with-out inverse kinematics (the IK algorithm calculates jointangles of a limb as the user positions the endpoint). It’salso possible to import motion capture data using oneof the standard formats.

As the user adds keyframes, their sequence can beviewed and edited in the score window, which is essen-tially a timeline. The partially composed piece can be ani-mated at any time in the stage window, using a variety ofbody representations including stick figures, boundingboxes, outlines, or mesh surfaces. Alternatively, the piececan be rendered without perceptible delay and played inthe 3D performance window.

A wide variety of choreographers—includingCunningham and Danny Lewis—have used this chore-ography package. Educators in K–12 and postsecondaryinstitutions also use it—particularly innovative educa-tional users include the states of Kentucky in the US andof New South Wales in Australia. While the tool hasmerit in its own right as an approach to thinking aboutmovement in 3D and across time, it helps with resourcesby letting students in more common computer labs plan

Tom CalvertSimon FraserUniversity

Lars WilkeCredoInteractive

Rhonda RymanUniversity ofWaterloo

Ilene FoxDance NotationBureau

Applications of Computers to Dance __________________

ApplicationsEditor: Mike Potelhttp://www.wildcrest.com

6 March/April 2005 Published by the IEEE Computer Society 0272-1716/05/$20.00 © 2005 IEEE

choreography that is later performed in scarce dancestudios. They can then make better use of the time spentwith the dancers.

DanceForms is suited as a tool in dance education,introducing students to a variety of dance movementsand allowing them to explore new ways to expressivelycombine those movements. Rhonda Ryman at theUniversity of Waterloo has created comprehensive con-tent including dictionaries for ballet (Ballet Moves II)and modern dance (Modern Dance Moves); both arepublished by Credo Interactive (http://www.danceforms.com). Ballet Moves II details hundreds ofpositions and steps as codified in the English andRussian schools of ballet, as well as select samples fromthe Italian school. Figure 2 shows an example. This con-

tent provides the teacher and the student with a richenvironment in which to explore dance animation.Complementing these modern dance and ballet movescontent collections, Ryman has published a tutorial-based practical guide to Dance Forms 1.0 (http://www.dcd.ca) offering a hands-on approach to learning theprogram with step-by-step exercises.

Composing and editing dance scoresThe Labanotation primer in the sidebar shows that a

score consists of symbols on a vertical staff. The symbol’smeaning depends on its shape and shading, verticallength, and placement in one of the staff columns. Severalword-processing-type editors support the compositionand editing of Labanotation scores. These include

IEEE Computer Graphics and Applications 7

Labanotation PrimerLabanotation is a system for analyzing and recording

human movement. The original inventor, Rudolf von Laban(1879–1958) was an important figure in European moderndance. He published this notation first in 1928 asKinetographie. Several people continued the developmentof the notation and Hutchinson Guest has written theclassic text.1 In the US, the New York-based DanceNotation Bureau supports Labanotation and archives manyLabanotation scores (http://www.dancenotation.org).

Each Labanotation symbol gives four pieces ofinformation. First, the symbol’s shape indicates thedirection of movement (see Figure A).

Next, the symbol’s shading shows the level of amovement; diagonal strokes for high, a dot for middle, andblackened for low (see Figure B).

Third, the symbol’s placement on the column on the staffindicates the part of the body that is moving. ALabanotation staff represents the human body; the center

line divides the left side of the body from the right. Symbolsto the left of the center line refer to the left-hand side of thebody, symbols to the right of the center line to the right-hand side of the body. Some body parts must be identifiedby a symbol, see Figure C.

Finally, the symbol’s length indicates duration of themovement. The staff is read from the bottom up; movingahead in time (see Figure D). The tick marks on the centerline divide the time into counts and the horizontal linescorrespond with the bar lines in the music. Movementswritten on the same horizontal line occur simultaneously;movements written one above another occur sequentially.Measure numbers and dancers’ counts appear to the left ofthe staff.

Reference1. A. Hutchinson Guest, Labanotation: The System of Analyzing

and Recording Movement, Taylor and Francis, 1987.

Left forward diagonal

Side rightSide left

Left back diagonal

Back

Right back diagonal

Right forward diagonal

Forward

Place

A Direction symbols.

(1) (2) (3)

B Symbol shading: (1) forward high, (2) place middle, and (3)right side low.

(1) (2) (4)(3)

C Symbols indicating body parts: (1) head, (2) face, (3) hands,and (4) front of left shoulder.

Left

arm

Left

han

d

Tors

o

Left

leg

gest

ure

Left

sup

port

Righ

t su

ppor

t

Righ

t le

g ge

stur

e

Upp

er b

ody

Righ

t ar

m

Righ

t ha

nd

Hea

d

Left side of body

Right side of body

D The staff.

■ AutoCAD-based Calaban, which is particularly use-ful for creating high-quality printed scores (http://www.bham.ac.uk/calaban/frame.htm);

■ AutoCAD-based Labanatory, which aims to let userssearch a score (http://www.labanatory.com/);

■ X-Windows-based Labanotation LED developed inAustralia (http://www-staff.it.uts.edu.au/~don/pubs/led.html);

■ MacBenesh for Benesh notation, developed by Rymanand her colleagues at the University of Waterloo andnow maintained and distributed by DanceWrite(http://members.rogers.com/dancewrite/index.html);and

■ Benesh Notation Editor developed by the BeneshInstitute in London for editing multistave notationscores (http://www.benesh.org/frames.html).

These editors work well, facilitating the writing andmore importantly the editing of scores. However, themost widely used editor is the LabanWriter (http://www.dance.ohio-state.edu/labanwriter). This

Macintosh-based 2D graphics editor specializes in creat-ing Labanotation scores (a Windows version is beingdeveloped). It was developed at the Dance Departmentof the Ohio State University by George Karl, ScottSutherland, and David Ralley under the direction of LucyVenable. LabanWriter lets the user create one or morestaffs on the virtual page and provides a palette ofLabanotation symbols that can be selected and placed onthe staff in the columns representing the appropriatebody part. Direction and turn symbols can be stretchedin the vertical direction to indicate their duration.Modifier symbols can be selected and placed in proximi-ty to the main symbols or columns that they modify. Thedevelopment of a score is illustrated in Figure 3.

LabanWriter treats symbols strictly as 2D graphi-cal objects that fall loosely into two subclasses:stretchable and fixed sized. Stretchable symbolshave associated time durations, such as directionsymbols and turn symbols, while fixed size symbolsare normally modifiers of columns, measures, orother symbols, or are used to annotate the score.

Applications

8 March/April 2005

1 DanceFormsprovides (a) a stagewindow forcomposingmultipledancers, (b) a studiowindow forcreating partic-ular body posi-tions, (c) a scorewindow toshow how eachdancer movesover time, and(d) a renderedperformancewindow.

(a) (b)

(c)

(d)

Column and timing information isnot explicitly stored with the sym-bols. The symbols are organizedinto columns and measures onlyinsofar as their Cartesian posi-tions indicate. LabanWriter hasno knowledge base for parsing thescore and nothing prevents theuser from entering complete non-sense.

The LabanWriter file (.lw4) is anASCII file that stores only enoughinformation to reproduce thearrangement of the graphical sym-bols just described. Each record inthe file contains the type, position,size, color, special fonts, and otherinformation related to the drawingof the symbols. The order of symbolswithin the data stream is the onelaid down by the user. No attempt ismade to sort the symbols in anyother way. (A detailed description ofthe file format is available athttp://www.dance.ohio-state.edu/labanwriter/LW4/LW44FileFormat.html). LabanWriter can also print the Labanotationscore, or produce a 2D raster image in one of three pop-ular graphic file formats (.png, .pict, or .jpeg).

LabanReader, also developed at the Ohio StateUniversity, is a software program for selective view-ing of notation symbols in a dance score. It can beused to focus on particular aspects of movement forteaching or to display various kinds of patterning formovement analysis. LabanReader is suited for usewith scores created in LabanWriter 4.0 or later.

As mentioned previously, LabanWriter and othernotation editors fulfill a useful function. However, todate, the developers have not accounted for knowledgeof the score’s structure (syntax or semantics). In con-trast, almost all word processors check spelling andmany make suggestions on grammar. The notation edi-tors so far have not applied rules to check the structureof what has been composed—they are merely drawingsystems for placing symbols on a staff. Another issue istheir inability to search a score for a particular move-ment pattern (Labanatory has some searching capabil-ities). One reason that notation editors are so limited isthat no consistent, unambiguous way exists of repre-senting human movement with a machine-readableontology or grammar of the kind that can be created inXML, for example.

Translating dance notation intoanimation

Not everyone in the dance community can read nota-tion, and writing dance scores is a specialized skill. Aprogram to translate notation into animation (and viceversa) could make the notation more accessible tochoreographers, dancers, and students. The New York-based Dance Notation Bureau, in collaboration withSimon Fraser University, University of Waterloo, and

Credo Interactive, has led the development ofLabanDancer, a program to translate LabanWriterscores into 3D animation.

Creating a composite scoreThe first stage in translating LabanWriter files is to

convert the stream of graphical symbols into a set ofcomposite data records that have meaning in theLabanotation context and to populate a data structurethat is a more direct analog of the Labanotation score.In this process, graphical symbols are spatially sortedand assigned to columns and measures based on the

IEEE Computer Graphics and Applications 9

2 Ballet Moves II provides a dictionary of ballet positions and movements, including thispalette of basic positions in the English school.

3 Composing a score with LabanWriter. The user selectsfrom the main menu (above left) to see pop-up menuscontaining families of symbols or signs (above right)that can be placed on the three-lined vertical staff.

symbol position relative to the staff origin. Modifier sym-bols are simultaneously collected and later associatedwith the symbols or columns that they modify based ontheir proximity (an inherently error-prone operation).

The composite score thus derived contains a map ofanimation channels; each channel is associated with adifferent limb segment or support (in most cases thiscorresponds to a column in the Labanotation score). Thedata records within each channel are ordered by time,and each record contains all possible modifier informa-tion. Ancillary information, such as timing, paths, orrelationships is also stored in the data structure outsideof the channels. In a final pass, the data record timing isadjusted to reflect the real timing of the gestures accord-ing to context-dependent rules.

Kinematic modelThe LabanDancer application uses a deformable,

polygonal mesh model driven by a hierarchical skele-ton. Keyframe animation channels control joint anglesin the skeleton, and four inverse kinematic chains drivethe arms, legs, feet, and hands. The software uses theIK using analytical methods (IKAN) algorithm, which isspecialized for controlling human-like limbs with 7degrees of freedom and has a deterministic rather thaniterative solution.2 IKAN also has the advantage of allow-ing control over the elbow or knee joints, which is impor-tant for achieving all possible poses specified by theLabanotation score. Keyframe channels control the endeffectors, representing foot and hand position and ori-entation. These channels can interpolate position eitherlinearly, or spherically, a necessary feature in creatingbelievable paths between poses. A 3D Bezier curve—theparameters of which a footsteps algorithm determines—controls the global position of the root of the hierarchy(that is, the pelvis).

Currently, LabanDancer provides four separate mod-els, two female and two male, customized for modernand classical dance. Users can easily create more mod-els representing different dance traditions by using com-mercially available modeling programs.

Three command typesThe composite score is parsed into three streams. The

first interprets gestures—these are non-weight-bearingmovements of any body part. The second interprets sup-port changes (including locomotion), and the thirdinvolves other issues such as repetition of a sequence,paths, use of floor plans, and so on.

While much of Labanotation is explicit—it objective-ly specifies the orientation of limb segments at a partic-ular time—there are numerous instances where theLabanotation is implicit—relying on notator and dancerknowledge to deduce the movement in the context ofthe piece in question. Thus, the translator program mustinclude a knowledge base from which movements canbe interpreted in their context.

Interpreting gesturesThe Labanotation symbols generally indicate unam-

biguously the start time, end time, and final orientationof limb parts involved in a gesture. They do not, howev-

er, explicitly specify the path to be followed in carryingout the gesture. The path can be deduced once the start-ing and ending orientations are known. LabanDanceruses an inverse kinematics algorithm and applies con-straints to ensure that articulated limbs carry out themovement in an appropriate plane. For some move-ments it’s necessary to add intermediate keyframes asadditional constraints to ensure that articulated limbsdo not move inappropriately, such as passing throughother body parts, for example.

The parser that interprets gestural commands is sim-ple at a high level—as noted, final limb orientations areusually explicitly specified. However, at a detailed levelthe parser is extremely complex since it must ensure thatevery movement is appropriate to the context. In addi-tion to the constraints discussed previously, other con-textual issues include the dance style—for example, afoot might be pointed in ballet, but not in folk dance—and the timing conventions used in Labanotation—although a gesture is written within a beat, the usuallyunderstood timing is for it to begin moving before thebeat and arrive on the beat. This can vary depending onthe context. For example, arm gestures must be syn-chronized with leg gestures, which in turn are con-strained by the support symbols—enough time must beleft to complete the leg gesture before it contacts theground for the next support, and a leg cannot gestureuntil weight is transferred to the other leg partwaythrough the step.

Interpreting support changesIn Labanotation the concept of support and support

change is the basis for all locomotion. The notationshows which limb supports the body over a period oftime and the direction (if any) of the movement. Supportchanges from foot to foot combined with forward move-ment, for example, result in a forward walk or run. Thusthe notation does not explicitly specify the flexion andextension of the limbs concerned, leaving it to the intel-ligent dancer to recognize which movements are neces-sary to achieve support change in a specific direction.

One approach to the animation of support changeswould be to store keyframe sequences for all possibletransitions; however, the number of sequences increas-es exponentially with the movement’s complexity.LabanDancer uses a general approach based on van dePanne’s footprints algorithm for animation of humanlocomotion.3 This algorithm calculates an optimum pathfor the center of gravity of a biped based on the place-ment and timing of the footsteps and the geometricalconfiguration. Once the path of the center of mass isknown, keyframes for the feet are generated, based onthe position and timing of footsteps and any restrictionsimposed on the flight phases of the foot. The footkeyframes drive the goal positions for the IK chains dri-ving the legs. Important in the choice of this algorithmwas the fact that it could take changes in level intoaccount, something that is important in Labanotation.Also, the algorithm seamlessly handles the transitionfrom walking to running, jumping, or hopping, and canhandle locomotion along a curved path.

To implement this approach it’s necessary to find a

Applications

10 March/April 2005

way to generate the timing and position of the footprintsfrom the composite Labanotation score. The footprintsalgorithm specifies the footsteps in absolute positionand relative (to the last) time, while Labanotation spec-ifies support changes in absolute time and relative (toplace) position. The translation from one system to theother is relatively straightforward.

Turns, pivots, and other commandsTurn symbols within the support columns of the

Labanotation score indicate a turn or pivot. The foot-steps algorithm that we use does not adequately takethis type of motion into account, and thus some modi-fications were required. Also, many other Labanotationsymbols do not translate into either gestures or changesin support. These include techniques to handle multi-ple dancers, the use of floor plans, symbols to indicaterepetition of a sequence, and so on. The parser recog-nizes these symbols and has procedures to implementthe corresponding movement.

LabanDancer prototypeThe first LabanDancer prototype has been imple-

mented for Windows and Mac OSX. Figure 4 shows theuser interface. Fairly standard controls allow the user toadjust the viewpoint by rotating the stage on all threeaxes and by providing for zooming in and out. In addi-tion to totally interactive control with the mouse, fixedviews can be chosen or the camera can follow thedancer. The display can show the dancer’s footprints,defining the path of the movement. Also, a sound out-put provides a metronome-like tock sound to indicateeach beat. An important interface feature is the displayof a simultaneous graphic of the original Labanotationscore on the left of the screen. A cursor moves up thescore as the animation progresses. The user can dragthe cursor on the score or on the timeline below the ani-mation to move backwards and forwards through theanimation sequence.

FutureThis human animation software currently translates

numerous types of movement including supports, ges-tures, turns, circular paths, and contracted limbs. Futureversions will be extended to handle all aspects of a scoreand in addition, LabanDancer will be written into a filethat can be read into DanceForms or other more gener-al programs for human figure animation. This file canthen be edited and combined in a variety of ways withother sequences from animation, motion capture, orLabanDancer translations of other Labanotation scores.A more complete description of LabanDancer can befound elsewhere.4

Current research and futuredevelopments

There are many things that can be done to enhanceand extend the computer technology used to animateand visualize dance, plan choreography, edit and ani-mate notation, and enhance performance. However, arecurring issue in many of these developments is theneed for a unique, unambiguous way to represent

human movement. Notation systems, such asLabanotation, do this at a conceptual level, but since thenotation itself does not have an unambiguous machinemachine-readable representation, the notation is notenough. The composite score data structure developedfor LabanDancer is a pragmatic attempt to address thisproblem. A more principled approach must start bydefining an ontology or grammar that captures the nec-essary syntax and semantics.

In April 2004, the Dance Notation Bureau and the OhioState University sponsored a workshop where 29 expertsin Labanotation, dance, human movement, and technol-ogy met to discuss adding intelligence to programs thatcompose, edit, and animate Labanotation. This wouldadd understanding of a symbol’s attributes, such as whichbody part is moving, timing, and so on, and enable newadvances in the use of technology and notation. (See themeeting report at http://dancenotation.org/DNB/news/report.html.) The consensus at the workshop wasthat there is a need for an interlingua—that is, an unam-biguous representation of movement that can map to dif-ferent notation systems and editors, animations, andmotion-captured dance data.

Like all efforts to develop international standards, thisis a challenging problem. Following the development ofMusicXML to provide structure for musical scores, therehas been some initial discussion of a possible LabanXMLas proposed by Minako Nakamura and her colleaguesin Japan5 or a more comprehensive DanceXML as pro-posed by Simon Fraser University graduate studentJonathan Hatol. It’s easy to see how XML can help cap-ture the high-level structure of Labanotation, but it’smore difficult to capture the more detailed context-dependent structure.

When a usable ontology-based representation is devel-oped, it will then be necessary to create interfaces to theexisting representations used in notation editors in sys-

IEEE Computer Graphics and Applications 11

4 LabanDancer interface lets the user follow the animation of aLabanotation score. Using the score or animation cursor the user can goback and forth, comparing the movement with the notation.

tems for animating dance and in motion capture systems.The great advantage of this new representation will bethat it will define all movement explicitly and not implic-itly, as with present systems. It will be possible to searcheasily for movement patterns and to use this power inchoreography, as well as many aspects of dance schol-arship. This development can also be useful in otherapplications that require the analysis of human move-ment such as animation for games or the representationof motion-captured human movement data used in theclinical diagnosis of locomotor problems. ■

References1. M. Cunningham, Changes/Notes on Choreography, F. Starr,

ed., Something Else Press, 1968.2. D. Tolani, A. Goswami, and N. Badler, “Real-Time Inverse

Kinematics Techniques for Anthropomorphic Limbs,”Graphical Models, vol. 62, no. 5, 2000, pp. 353-388;http://hms.upenn.edu/software/ik/ik.html.

3. M. Van de Panne, “From Footprints to Animation,” Com-puter Graphics Forum, vol. 16, no. 4, 1997, pp. 211-223.

4. L. Wilke et al., “Animating the Dance Archives,” Proc. 4th Int’lSymp. Virtual Reality, Archaeology and Intelligent CulturalHeritage (VAST), Eurographics Assoc., 2003, pp. 91-99.

5. M. Nakamura, “Text Representation of Labanotation Datafor Computer Based Motion Analysis,” presented at theWorld Dance Assoc./Int’l Council of Kinetography Laban/Congress on Research in Dance Int’l Conf., 2004; http://www.imb.is.ritsumei.ac.jp/~hachi/hachi_e.html.

Readers may contact Tom Calvert at [email protected].

Readers may contact Mike Potel at [email protected].

Applications

12 March/April 2005

Computer Graphics and InteractiveTechnologies in Live Performance

There are many examples of the use of all formsof multimedia in conjunction with live danceperformances. In some cases a preplannedanimation sequence or digital video withappropriate sound is played as a backdrop and livedancers interact with the preprogrammed display.A more technically challenging system is requiredwhen it senses the live dancers’ movements andmodifies the imagery and/or sound. Anothersituation arises when digital video, possibly withimmersive or stereo display, links two distantlocations for rehearsal and/or performance. Allkinds of other variation are possible.

One ambitious example is the Trajets(http://www.trajets.net/) interactive installationcodirected by Susan Kozel and Gretchen Schiller,with development by Rob Lovell and Scott Wilson.Here, 12 trapezoid-shaped, motorized screens are

suspended from a grid and spiral in response tovisitors. Vision-sensing-software eyes let thecomputers localize the visitors’ positions in space.Different visual sequences are projected on each ofthe screens. The visuals, which experiment withbody movement, have a unique digital look andfeel that the artists created in the video shootingprocess. One image captures a person slowlyjumping and falling and other images show bodiesin mud and water—Figure A gives a sense of thiscomplex environment.

Another exploratory project is Whispers(http://whisper.surrey.sfu.ca/index2.html) byThecla Schiphorst and others, which involvesphysiological sensors, small wearable computingdevices, wireless computer communication, andhandheld technologies embedded in evocativeand playful garments worn by the participants. Itmaps physiological data onto linked andnetworked devices worn on or close to the skinand in garments. In other words, it involvescollecting data from the bodies of participants andcommunicating the participants’ responses to thesystem and other participants throughvisualization and sonification techniques.

One venue where much of the latest work isdisplayed is the biannual Monaco Dance Forum(http://www.monacodanceforum.com). Theforum is an international dance meeting andfestival that offers dance performances,multimedia installations, exhibitions, video dancescreenings, and symposiums.

Another source for work up to 2000 is theDigital Performance Archive (http://dpa.ntu.ac.uk/dpa_site/).

A View of the Trajets interactive installation.

Cou

rtes

y Ro

th-K

imur

a A

rchi

tect

s, H

olua

loa,

HI