Dance-the-Music: an educational platform for the modeling, recognition and audiovisual monitoring of dance steps using spatiotemporal motion templates

RESEARCH Open Access

Dance-the-Music: an educational platform for themodeling, recognition and audiovisualmonitoring of dance steps using spatiotemporalmotion templatesPieter-Jan Maes*, Denis Amelynck and Marc Leman

Abstract

In this article, a computational platform is presented, entitled “Dance-the-Music”, that can be used in a danceeducational context to explore and learn the basics of dance steps. By introducing a method based onspatiotemporal motion templates, the platform facilitates to train basic step models from sequentially repeateddance figures performed by a dance teacher. Movements are captured with an optical motion capture system. Theteachers’ models can be visualized from a first-person perspective to instruct students how to perform the specificdance steps in the correct manner. Moreover, recognition algorithms-based on a template matching method-candetermine the quality of a student’s performance in real time by means of multimodal monitoring techniques. Theresults of an evaluation study suggest that the Dance-the-Music is effective in helping dance students to masterthe basics of dance figures.

Keywords: dance education, spatiotemporal template, dance modeling and recognition, multimodal monitoring,audiovisual dance performance database, dance-based music querying and retrieval

1 IntroductionThrough dancing, people encode their understanding ofthe music into body movement. Research has shownthat this body engagement has a component of temporalsynchronization but also becomes overt in the spatialdeployment of dance figures [1-5]. Through dancing,dancers establish specific spatiotemporal patterns (i.e.,dance figures) in synchrony with the music. Moreover,as Brown [1] points out, dances are modular in organi-zation, meaning that the complex spatiotemporal pat-terns can be segmented into smaller units, calledgestures [6]. The beat pattern presented in the musicfunctions thereby as an elementary structuring element.As such, an important aspect of learning to dance islearning how to perform these basic gestures inresponse to the music and how to combine these ges-tures to further develop complex dance sequences.

The aim of this article is to introduce a computationalplatform, entitled “Dance-the-Music”, that can be usedin dance education to explore and learn the basics ofdance figures. A special focus thereby lays on the spatialdeployment of dance gestures, like footstep displace-ment patterns, body rotation, etc. The platform facili-tates to train basic step models from sequentiallyrepeated dance figures performed by a dance teacher.The models can be stored together with the correspond-ing music in audiovisual databases. The contents ofthese databases, the teachers’ models, are then used (1)to give instructions to dance novices on how to performthe specific dance gestures (cf., dynamic dance notation),and (2) to recognize the quality of students’ perfor-mances in relation to the teachers’ models. The Dance-the-Music was designed explicitly from a user-centeredperspective, meaning that we took into account aspectsof human perception and action learning. Four impor-tant aspects are briefly described in the following para-graphs together with the technologies we developed toput these aspects into practice.

* Correspondence: [email protected], Department of Musicology, Ghent University, Blandijnberg 2, 9000Ghent, Belgium

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

© 2012 Maes et al; licensee Springer. This is an Open Access article distributed under the terms of the Creative Commons AttributionLicense (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium,provided the original work is properly cited.

Spatiotemporal approachWhen considering dance gestures, time-space dependen-cies are core aspects. This implies that the spatialdeployment of body parts is directly linked to the tem-poral structure outlined in the music (involving rhythmand timing). The modeling and automatic recognition ofdance gestures often involve Hidden Markov Modeling(HMM) [7-10]. However, HMM has the property toexhibit some degree of invariance to local warping(compression and stretching) of the time-axis [11]. Eventhough this might be an advantage for applications likespeech recognition, it is a serious drawback when con-sidering spatiotemporal relationships in dance gestures.HMMs are fine for detecting basic steps and spatial pat-terns but cause major difficulties for timing aspectsbecause of the inherent time-warping mechanism.Therefore, for the Dance-the-Music, we will introducean approach based on spatiotemporal motion templates[12-14]. As will be explained in depth, the discrete timesignals representing the gestural parameters extractedfrom dance movements are organized into a fixed-sizemultidimensional feature array forming the spatiotem-poral template. Dance gesture recognition will beachieved by a template matching technique based oncross-correlation computation.

User- and body-centered approachThe Dance-the-Music facilitates to instruct dance ges-tures to dance novices with the help of an interactivevisual monitoring aid (see Sections 3.4.1 and 4). Con-cerning the visualization of basic step models, we takeinto account two aspects involving the perception andunderstanding of complex multimodal events, like dancefigures. First, research has shown that segmentation ofongoing activity into smaller units is an automatic com-ponent of human perception and functional for memoryand learning processes [1,15]. For this, we applied algo-rithms that segment the continuous stream of motioninformation into a concatenation of elementary gestures(i.e., dance steps) matching the beat pattern in themusic (cf., [6]). Each of these gestures is conceived as aseparate unit, having a fixed start- and endpoint. Sec-ond, neurological findings indicate that motor represen-tations based on first-person perspective action involve,in relation to a third-person perspective, more kines-thetic components and take less time to initiate thesame movement in the observer [16]. Although applica-tions in the field of dance gaming and education oftenenable a manual adaptation of the viewpoint perspective,they do not follow automatically when users rotate theirbody during dance activity [17-20]. In contrast, thevisual monitoring aid of the Dance-the-Music automati-cally adapts the viewpoint perspective in function of therotation of the user at any moment.

Direct, multimodal feedbackThe most commonly used method in current dance edu-cation to instruct dance skills is the demonstration-per-formance method. As will explained in Section 2, theDance-the-Music elaborates on this method in thedomain of human-computer interaction (HCI) design. Inthe demonstration-performance method, a model perfor-mance is shown by a teacher which must then be imi-tated by the student under close supervision. As Hoppeet al. [21] point out, a drawback to this learning sche-matic is the lack of an immediate feedback indicatinghow well students use their motor apparatus in responseto the music to produce the requisite dance steps. Studieshave proven the effectiveness of self-monitoring throughaudiovisual feedback in the process of acquiring dancingand other motor skills [19,22-24]. The Dance-the-Musictakes this into account and provides direct, multimodalfeedback services. It is in this context that the recognitionalgorithms–based on template matching–have their func-tionality (see Section 3.3). Based on cross-correlationcomputation, they indicate how well a student’s perfor-mance of a specific dance figure matches the correspond-ing model of the teacher.

Dynamic, user-oriented frameworkThe Dance-the-Music is designed explicitly as a compu-tational framework (i.e., a set of algorithms) of whichcontent and configuration settings are entirely depen-dent on the needs and wishes of the dance teacher andstudent. The content mainly consists of the dance fig-ures that the teacher wants to instruct to the studentand the music that corresponds with it. Configurationsettings involve tempo adjustment, the number of stepsin one dance figure, the number of cycles to perform totrain a model, etc. Moreover, the Dance-the-Music isnot limited to the gestural parameters presented in thisarticle. Basic programming skills facilitate to input dataof other motion tracking/sensing devices, extract otherfeatures (acceleration, rotational data of other bodyparts, etc.), and add these into the model templates.This flexibility is an aspect that distinguishes the Dance-the-Music from commercial hardware (e.g., dance dancerevolution [DDR] dancing pad interfaces) and softwareproducts (e.g., StepMania for Windows, Mac, Linux;DDR Hottest Party 3 for Nintendo Wii; DanceDanceRe-volution for PlayStation 3, DDR Universe 3 forXbox360, Dance Central and Dance Evolution forKinect, etc.). Most of these systems use a fixed, built-invocabulary of dance moves and music. Another majordownside to most of these commercial products is thatthey provide only a small action space restricting spatialdisplacement, rotation, etc. The Dance-the-Music drasti-cally expands the action/dance space facilitating rota-tion, spatial displacement, etc.

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 2 of 16

The structure of the article is as follows: In Section 2,detailed information is provided about the methodologi-cal grounds on which the instruction method of theeducational platform is based. Section 3 is then dedi-cated to an in-depth description of the technological,computational, and statistical aspects underlying thedesign of the Dance-the-Music application. In Section 4,we present a user study conducted to evaluate if the sys-tem can help dance novices in learning the basics ofspecific dance steps. To conclude, we discuss in Section5 the technological and conceptual performance andfuture perspectives of the application.

2 Instruction methodIn concept, the Dance-the-Music brings the traditionaldemonstration-performance approach into the domainof HCI design (see Section 1). Although the basic

procedure of this method (i.e., teacher’s demonstration,student’s performance, evaluation) stays untouched, theintegration of motion capture and real-time computerprocessing drastically increase possibilities. In whatcomes, we outline the didactical procedure incorporatedby the Dance-the-Music in combination with the tech-nology developed to put it into practice.

2.1 Demonstration modeA first mode facilitates dance teachers to train basic stepmodels from their own performance of specific dancefigures. Before the actual recording, the teacher is ableto configure some basic settings, like the music onwhich to perform, the tempo of the music, the numberof steps per dance figure, the amount of training cycles,etc. (see module 1 and 2, Figure 1). Then, the teachercan record a sequence of a repetitive performed dance

Figure 1 Graphical user interface (GUI) of the Dance-the-Music.

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 3 of 16

figure of which the motion data is captured with opticalmotion capture technology (see module 3, Figure 1).When the recording is finished, the system immediatelyinfers a basic step model from the recorded trainingdata. The model can then be displayed (module 4, Fig-ure 1) and, when approved, stored in a databasetogether with the corresponding music (module 5, Fig-ure 1). This process can then be repeated to create alarger audiovisual database. These databases can besaved as .txt files and loaded whenever needed.

2.2 Learning (performance) modeBy means of a visual monitoring aid (see Figure 2, left)with which a student can interact, the teachers’ modelscan be graphically displayed from a first-person perspec-tive and can be segmented into individual steps. By imi-tating the graphically notated displacement and rotationpatterns, a dance student learns how to perform thestep patterns in a proper manner. In order to supportthe dance novice, the playback speed of the dynamicvisualization is made variable. When played in the origi-nal tempo, the model can be displayed in synchronywith the music that corresponds with it. Moreover,recognition algorithms are implemented facilitating acomparison between the model and the performance ofthe dance novice (see Section 3.3). As such, direct mul-timodal feedback can be given monitoring the quality ofa performance (see Section 3.4).

2.3 Gaming (evaluation) modeOnce students learned to perform the dance figures withthe visual monitoring aid, they can exhibit their danceskills. This is the application mode allowing students toliterally “Dance the Music”. By performing a specificdance figure learned with the visual monitoring aid, stu-dents receive music that fits a particular dance genre. Itis in this context of gesture-based music retrieval thatthe recognition algorithms based on template matchingcome to the fore (see Section 3.3). Based on cross-corre-lation computation, these algorithms detect how exact aperformed dance figure of a student matches the modelperformed by the teacher. The quality of the student’sperformance in relation to the teacher’s model is thenexpressed in the auditory feedback and in a numerical

score stimulating the student to improve his/herperformance.The computational platform itself is built in Max/MSP

http://www.cycling74.com. The graphical user interface(GUI) can be seen in Figure 1. It can be shown on anormal computer screen or projected on a big screen oron the ground. One can interact with the GUI with acomputer mouse. The design of the GUI is kept simpleto allow intuitive and user-friendly accessibility.

3 Technical designDifferent methods are used for modeling and recognizingmovement (e.g., HMM-based, template-based, state-based,etc.). For the Dance-the-Music, we have made the deliber-ate choice to implement a template-based approach togesture modeling and recognition. In this approach, thediscrete time signals representing the gestural parametersextracted from dance movements are organized into afixed-size multidimensional feature array forming the spa-tiotemporal template. For the recognition of gestures, wewill apply a template matching technique based on cross-correlation computation. A basic assumption in thismethod is that gestures must be periodic and have similartemporal relationships [25,26]. At first sight, HMMs ordynamic time warping (DTW)-based approaches might beunderstood as proper candidates. They facilitate learningfrom very few training samples (e.g., [27,28]) and a smallnumber of parameters (e.g., [29]). However, HMM andDTW-based methods exhibit some degree of invariance tolocal time-warping [11]. For dance gestures in whichrhythm and timing are very important, this is problematic.Therefore, when explicitly taking into account the spatio-temporal relationship of dance gestures, the template-based method we introduce in this article provides us witha proper alternative.In the following sections, we first go into more detail

how dance movements are captured (Section 3.1). After-wards, we will explain how the raw data is pre-pro-cessed to obtain gestural parameters which areexpressed explicitly from a body-centered perspective(Section 3.1.2). Next, we will point out how the Dance-the-Music models (Section 3.2) and automatically recog-nizes (Section 3.3) performed dance figures using spatio-temporal templates and how the system providesaudiovisual feedback of this performance (Section 3.4).A schematic overview of Section 3 is given in Figure 3.

3.1 Motion capture and pre-processing of movementparametersMotion capture is done with an infrared (IR) optical sys-tem (OptiTrack/Natural Point). Because we are inter-ested in the movements of the body-center and feet, weattach rigid bodies to these body parts (see Figure 4).The body-center (i.e., center-of-mass) of a human body

Mocap

section 3.1

Pre-processing

section 3.1

Modeling

section3.2 Recognition + feedback

section 3.3 and 3.4

Audiovisual monitoring

section 3.4

Figure 2 Schematic overview of the technical design of theDance-the-Music.

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 4 of 16

in standing position is situated in the pelvic area (i.e.,roughly the area in between the hips). Because visualocclusion can occur (with resulting data loss) when thehands cover hip markers, it can be opted to attach themto the back of users instead (see Section 3.1.2, par. Spa-tial displacement). A rigid body consists of minimum

three IR-reflecting markers of which the mutual distanceis fixed. As such, based on this geometric relationship,the motion capture system is able to identify the differ-ent rigid bodies. Furthermore, the system facilitates tooutput (1) the 3-D position of the centroid of a rigidbody, and (2) the 3-D rotation of the plane formed bythe three (or more) markers. Both the position and rota-tion components are expressed in reference to a globalcoordinate system predefined in the motion capturespace (see Figure 5). These components will be referredto as absolute, in contrast to their relative estimates inreference to the body (see Section 3.1.1).For the Dance-the-Music, the absolute (x, y, z) values

of the feet and body-center together with the rotation ofthe body-center expressed in quaternion values (qx, qy,qz, qw) are streamed, using the open sound control(OSC) protocol to Max/MSP at a sample rate of 100 Hz.3.1.1 Relative position calculationThe position and rotation values of the rigid body definedat the body-center are used to transform the absoluteposition coordinates into relative ones in reference to abody-fixed coordinate system with an origin positionedat the body-center (i.e., local coordinate system). Theposition and orientation of that local coordinate systemin relation to the person’s body can be seen in moredetail in Figure 5. The transformation from the initialbody stance (Figure 5, left) is executed in two steps. Bothare incorporated in real-time operating algorithms,implemented in Max/MSP as java-coded mxj-objects.

1. Rotation of the local, body-fixed coordinate sys-tem in a way it has the same orientation as the glo-bal coordinate system (Figure 5, middle). Whatactually happens, is that all absolute (x, y, z) valuesare rotated based on the quaternion values of therigid body attached to the body-center representingthe difference in orientation between the local andthe global coordinate system.2. Displacement of the origin (i.e., body-center) ofthe local, body-fixed coordinate system to the originof the global coordinate system (Figure 5, right).

As such, all position values can now be interpreted inreference to a person’s own body-center. However, a

Figure 3 Placement of the rigid bodies on the dancer’s body.

Figure 4 Representation of how the body-fixed local coordinate system is translated to coincide with the global coordinate system.

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 5 of 16

problem inherent to this operation is that rotations ofthe rigid body attached to the body-center, independentfrom actual movement of the feet, do result in apparentmovement of the feet. The consequences for free move-ment (for example for the upper body) are minimalwhen taking into account a well-considered placementof the rigid body attached to the body-center. The pla-cement of the rigid body at the hips, as shown in Figure4, does not constrain three-dimensional rotations of theupper body. However, the problem remains for particu-lar movements in which rotations of the body-centerother than the rotation around the vertical axis areimportant features, like lying down, rolling over theground, movements where the body-weight is (partly)supported by the hands, flips, etc. Apart from the pro-blems they cause for the mathematical procedures pre-sented in this section, these movements are alsoincompatible with the visualization strategy which is dis-cussed into more detail in Section 3.4.1. As such, thesemovements are out of the scope of the Dance-the-Music.3.1.2 Pre-processing of movement parametersAs already mentioned in the introduction, the first stepin the processing of the movement data is to segmentthe movement performance into discrete gestural units(i.e., dance steps). The borders of these units coincidewith the beats contained in the music. Because theDance-the-Music requires music to be played at a stricttempo, it is easy to calculate where the (BPs) are situ-ated. The description of the discrete dance steps itself is

aimed towards the spatial deployment of gestures per-formed by the feet and body-center. The descriptioncontains two components: First, the spatial displacementof the body-center and feet, and second, the rotation ofthe body around the vertical axis.Spatial displacement This parameter describes thetime-dependent displacement (i.e., spatial segment) ofthe body-center and feet from one beat point (i.e., BPbe-gin) to the next one (i.e., BPend) relative to the posturetaken at the time of BPbegin. With posture, we indicatethe position of the body-center and both feet at a dis-crete moment in time. Moreover, this displacement isexpressed with respect to the local coordinate system(see Section 3.1.1) defined at BPbegin. In general, thealgorithm executes the calculation in three steps:

1. Input of absolute (x, y, z) values of body-centerand feet at a sample rate of 100 Hz.2. Calculation of the (x, y, z) displacement relative tothe posture taken at BPbegin expressed in the globalcoordinate system (see Equation 1):® For this, at the beginning of each step (i.e., ateach BPbegin), we take the incoming absolute (x, y, z)value of the body-center and store it for the com-plete duration of the step. At each instance of thesteptrajectory that follows, this value is subtractedfrom the absolute position values of the body-center,left foot, and right foot. This operation places thebody-center at each BPbegin in the middle of the glo-bal coordinate system. As a consequence, this “reset”operation results in jumps in the temporal curvesforming separate spatial segments correspondingeach to one dance step (e.g., Figure 6, bottom). Thedisplacement from the posture taken at each BPbeginis still expressed in an absolute way (i.e., withoutreference to the body). Therefore, the algorithmneeds to perform a final operation.3. Rotation of the local coordinate system in a way ithas the same orientation as the global coordinatesystem at BPbegin (cf., Section 3.1.1, step 1):® Similar to the previous step, only the orientationof the rigid body attached to the body-center at eachnew BPbegin is taken into account and used succes-sively to execute the rotation of all the followingsamples belonging to the segment of a particularstep.4. Calibration:® Before using the Dance-the-Music, a user is askedto take a default calibration pose, meaning to standup straight with both feet next to each other. The(x, y, z) values of the feet obtained from this poseare stored and used to subtract from the respectivecoordinate values of each new incoming sample. Assuch, the displacement of the feet is described at

Figure 5 Top left: m×n×p template storing the training data.Each cube consists of one numeric value which is a function of thetime, gestural parameter and sequence. Top right: m×n templaterepresenting a basic step model. Bottom left: The five linesrepresent an example of the contents of the gray cubes in the topleft template (with n = 800, and p = 5). Bottom right:Representation of the discrete values stored in the gray featurearray in the top right template.

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 6 of 16

each moment in time in reference to that pose. Thiscalibration procedure enables to compensate for (1)individual differences in leg length, and (2) changesin the placement of the rigid bodies correspondingto the body-center. As such, one can opt to placethat rigid body somewhere else on the torso (seeFigure 2).

(�x, �y, �z)[BPi ,BPi+1[=(x,y,z)−(x,y,z)BPi (1)

Rotation According to Euler’s rotation theorem, any 3-D displacement of a rigid body whereby one point ofthe rigid body remains fixed, can be expressed as a sin-gle rotation around a fixed axis crossing the fixed pointof the rigid body. Such a rotation can be fully definedby specifying its quaternions. A quaternion representa-tion of a rotation is written as a normalized four-dimen-sional vector [qx qy qz qw]

T , linked to the rotation axis[ex ey ez]

T and rotation angle ψ.In Section 3.1.1, we outlined the reasons why the rota-

tion of the rigid body attached to the body-center isrestricted to rotations around the vertical axis withouthaving too severe consequences for the freedom ofdance performances. This is also an important aspectwith respect to the calculation of the rotation aroundthe vertical axis departing from quaternion values. Everyrotation, expressed by its quaternion values, can then beapproximated by a rotation around the vertical axis [0 0± 1]T or in aeronautics terms rotations are limited toyaw. Working with only yaw gives us the additionalbenefit of being able to split-up a dance movement in achain of rotations where every rotation is specified withrespect to the orientation at the beginning of each step(i.e., at each BP). The calculation procedure consists oftwo steps:

1. Calculation of the rotation angle around the verti-cal axis:

® The element qw in the quaternion (qx, qy, qz, qw)of the rigid body attached to the body-center deter-mines the rotation angle ψ (qw = cos(ψ/2). We usethis rotation angle as an approximation value for therotation angle around the vertical axis (i.e., yawangle Ψ). Implicitly, we suppose that the values forqx and qy are small meaning that the rotation axisapproximates the vertical axis: [ex ey ez]

T = [0 0 ± 1]T.2. Calculation of the rotation angle relative to theorientation at BPbegin (see Equation 2):® The method to do this is similar to the onedescribed in the second step of the previous para-graph (’Spatial displacement’).

��[BPi ,BPi+1] = � − �BPi (2)

3.2 Modeling of dance figuresIn this section, we outline how we apply a template-based approach for modeling a sequence of repetitivedance figures performed on music. The parameters ofthe–what we will call–basic step model are the onesdescribed in Section 3.1.2, namely the relative displace-ments of the body-center and feet, and the relative rota-tion of the body in the transverse plane per individualdance step.The basic step model is considered as a spatiotemporal

representation indicating the spatial deployment of ges-tures with respect to the temporal beat pattern in themusic. The inference of the model is conceived as asupervised machine-learning task. In supervised learning,the training data consists of pairs of input objects and adesired output value. In our case, the training data con-sists of a set of p repetitive cycles of a specific dance fig-ure of which we process the gestural parameters asexplained in Section 3.1.2. The timing variable is theinput variable and the gestural parameters are the desiredvalues. The timing variable depends on (1) the number ofsteps per dance figure, (2) the tempo in which the stepsare performed, and (3) the sample rate of the incomingraw movement data, according to Equation 3.

n =60 ∗ Steps per Figure*sample rate (Hz)

tempo (bpm)(3)

As such, the temporal structure of each cycle isdefined by a fixed number of samples (i.e., 1 to n). Theresult is a single, fixed-size template of dimensionm×n×p, with m equal to the number of gestural para-meters (cf., Section 3.1.2), n equal to the number ofsamples defining one dance figure (cf., Equation 3), andp equal to the number of consecutive cycles performedof the same dance figure (see Figure 6).

Figure 6 Template matching schematic.

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 7 of 16

To model each of the gestural parameters, we use adedicated K-Nearest Neighbor regression calculated withL1 loss function. In all these models, time is the regres-sor. The choice for an L1 loss function (L1 = |Y - f(t))|)originates in its robustness (e.g., protection against dataloss, outliers, etc.). In this case the solution is the condi-tional median, f(t) = median(Y |T = t) and its estimatesare more robust compared to an L2 loss function solu-tion that reverts to the conditional mean [[30], p. 19-20]. We calculate the median of the displacement valuesand rotation value located in the neighborhood of thetimestamp we want to predict for. Since we have a fixednumber of sequences per timestamp (i.e., p) a logicalchoice is to choose all these values for nearest neighborselection. The “K” - in the K-nearest neighbor selectionis then determined by the number of sequences per-formed of the dance figure. The model that eventuallywill be stored as reference model consists of an array ofvalues, one for each timestamp (see Figure 6).Because the median filtering is applied sample per

sample, it results in “noisy” temporal curves. Tests haveproven that smoothing the temporal curves stored inthe template improve the results of the recognition algo-rithms described in Section 3.3. Therefore, we smooththe temporal curves of the motion parameters of themodel template with a Savitzky-Golay FIR filter (cf.,[31]). This is done segment per segment to preserve the“reset” operation applied during the processing of themotion parameters (see Section 3.1.2). This type ofsmoothing has the advantage of preserving the spatialcharacteristics of the original data, like widths andheights, and it is also a stable solution.The system is now able to model different dance fig-

ures performed on specific musical pieces and, subse-quently, to store the basic step models in a databasetogether with the corresponding music. In what follows,we will refer to these databases as dance figure/musicdatabases. One singular database is characterized bydance figures which consist of an equal amount of dancesteps performed at the same tempo. However, as manydatabases as one pleases can be created varying withrespect to the amount of dance steps and tempi. Thesedatabases can then be stored as .txt files and loaded againafterwards. Once a database is created, it becomes possi-ble to (1) visualize the basic step models contained in it,and (2) compare a new input dance performance withthe stored models and provide direct audiovisual feed-back on the quality of that performance. These featuresare described in the remaining part of this section on thetechnical design of the Dance-the-Music.

3.3 Dance figure recognitionThe recognition functionalities of the Dance-the-Musicare intended to estimate the quality of a student’s

performance in relation to a teacher’s model. It is theexplicit goal to help students to learn to imitate the tea-chers’ basic step models as closely as possible. There-fore, the recognition algorithms are implemented toprovide a measure of similarity (for individual motionfeatures or for the overall performance). This measure isthen used to give students feedback about the quality oftheir performance. For example, the dance-based musicretrieval service presented in Section 3.4.2 must be con-ceived from this perspective.In this section, we outline the mathematical method

for estimating in real time the similarity between newmovement input and basic step models stored in adance figure/music database. For this, we will use a tem-plate matching method. This means that the gesturalparameters calculated from the new movement inputwill be stored in a single, fixed-size buffer template,which can then be matched with the templates of thestored models (see Figure 7). A crucial requirement ofsuch a method is that it must compensate for smalldeviations from the model in space as well as in time(cf., [32]). Spatial deviations do not necessarily need tobe considered as errors. A small deviation in space(movement is performed slightly more to the left orright, higher or lower, forward or backward) should notbe translated into an error. Similar, a performanceslightly scaled with respect to the model (bigger orsmaller) should also not be considered as an error.using normalized root mean square error (NRMSE) as ameans to measure error is not appropriate as it doespunish spatial translation and scaling errors. A better

Figure 7 Example of the internal mechanism of the templatematching algorithm. It represents the result of the comparison ofa dance figure consisting of eight steps (defined each by 100samples) performed by a student (here, subject 8 of the user studypresented in Section 4) against all stored models (N = 9) at eachBPbegin.

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 8 of 16

indicator for our application is the Pearson product-moment correlation coefficient r. It measures the sizeand direction of the linear relationship between our twovariables (input and model). A perfect performancewould result in a correlation coefficient that is equal to1, while a total absence of similarity between input ges-ture and model would lead to a correlation coefficientof 0. Timing deviations are compensated by calculatingthe linear relationship between the gestural input andmodel as a function of a time-lag (cf., cross-correlation).If we apply a time-lag window of i samples in bothdirections, then we obtain a vector of i+1 r values. Themaximum value is then chosen and outputted as corre-lation coefficient for this model together with the corre-sponding time-lag. As such, we obtain an objectivemeasurement of whether a dance performance antici-pates or is delayed with respect to the model.The buffer consists of a single, fixed-size template of

dimension m×n, with m equal to the number of gesturalparameters (cf., Section 3.1.2), and n equal to the num-ber of samples defining one dance figure (cf., Equation3). When a new sample - containing a value for eachprocessed gestural parameter - comes in, the systemneeds a temporal reference indicating where to store thesample in the template buffer on the Time axis. For this,dance figures are performed on metronome ticks follow-ing a pre-defined beat pattern and tempo. As such, itbecomes possible to send a timestamp along with eachincoming sample (i.e., a value between 1 and n).Because the buffer needs to be filled first, an input can

only be matched properly to the models stored in a

dance figure/music database after the performance of thefirst complete dance figure. From then on, the systemwill compare the input buffer with all the models at theend of each singular dance step. This results for eachmodel in m r values, with m corresponding to the num-ber of different parameters defining the model. Fromthese m values, the mean is calculated and internallystored. Once a comparison with all models is made, thehighest r value is outputted together with the number ofthe corresponding model. An example of this mechanismis shown in Figure 8. The dance figure/dance database ishere filled with nine basic step models. From these ninemodels, the model corresponding with the r values indi-cated with thicker line width, is the model that at alltimes most closely relates to the dance figure of whichthe data is stored in the input buffer template. As such,this would be the correlation coefficient that is outputtedby the system together with the model number.

3.4 Audiovisual monitoring of the basic step models andreal-time performancesAs explicated in Section 2, multimodal monitoring ofbasic step models and real-time performances is animportant component of the Dance-the-Music. In thefollowing two sections, we explain in more detail respec-tively the visual and auditory monitoring features of theDance-the-Music.3.4.1 Visual monitoringThe contents of the basic step models can be visuallydisplayed (see Figure 9) as a kind of dynamic and real-time dance notation system. What is displayed is (1) the

Figure 8 Visualization of the visual monitoring aid interface.

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 9 of 16

spatial displacement of the body-center and feet, and (2)the rotation of the body around the vertical axis fromBPbegin to BPend. The visualization is dynamic in the wayit can be played back in synchronization with the musicon which it was originally performed. It is also possibleto adapt the speed of the visual playback (but then,without sound). The display visualizes each dance stepof a basic step model in a separate window. Figure 9shows the graphical notation of an eight-step basicsamba figure as performed by the samba teacher of theevaluation experiment presented in Section 4. The win-dow at the left visualizes the direct feedback that usersget from their own movement when imitating the basicstep model represented in the eight windows placed atthe right. On top of the figure, one can see the maininterface for controlling the display features. The mainsettings involve transport functions (play, stop, reset,etc.), tempo settings, and body part selection.The intent is to visualize the displacement patterns (i.e.,

spatial segments) of each step on a two-dimensionalplane which represents the ground floor on which thedance steps were performed (see Figure 9). In otherwords, the displacement patterns are displayed on theground plane and viewed from a top-view perspective.Altering the size of the dots of which the trajectoriesexist, enable us to visualize the third, vertical dimensionof the displacement patterns. The red dots and purpletrajectories define the displacement patterns of the rightfoot, the green dots and yellow trajectories the ones ofthe left foot, and the black dots and trajectories the onesof the body-center. The vague-colored dots represent theconfiguration of the feet and body-center relative to eachother at the beginning of the step (BPbegin, the sharp-colored dots the configuration and the end of the step(BPend). As can be seen, as a result of the segmentationprocedure presented in Section 3.1.2, the position of thebody-center is reset at each new BPbegin. The triangleindicates the orientation of the body around the verticalaxis. Moreover, the orientation of the windows (and allthe data visualized in it) needs to be understood in

reference to the local reference frame of the dancer (seeFigure 5). Initially, the orientation and positioning ofeach window with respect to the local frame is as indi-cated by the XY coordinate system visualized in the leftwindow. However, when dance novices are using thevisual monitoring aid, they can make the orientation ofthe movement patterns of the basic step model displayedin each window dependable on their own rotation at thebeginning of each new step. This means that the XYcoordinate system (and, with that, all data visualizing themodel) is rotated in such a way that it coincides with thelocal frame of the dance novice. As such, the basic stepmodel is visualized at each instance from a first-personperspective. This way of displaying information presentsan innovative way of giving real-time instructions abouthow to move the body and feet to perform a step prop-erly. Now, this information can be transferred to the dan-cer in different ways:

1. The most basic option is to display the interfaceon a screen or to project it onto a big screen. Whena dance figure involves a lot of turns around the ver-tical axis, it is difficult to follow the visualization andfeedback on the screen. An alternative displaymethod provides a solution to this problem. It con-cerns the projection of the displacement informationdirectly on the ground. We used this last approachin the evaluation study presented in Section 4 (seeFigure 2).2. An alternative method projects the windows oneby one, instead of all eight windows at once (see Fig-ure 10). The position and rotation of the window isthereby totally dependent of the position and rota-tion of the user at the beginning of each new dancestep (BPbegin). A new window is then projected ontothe ground at each BPbegin, as such that the centroidof the window coincides with the position taken bythe person at that moment. The rotation of the win-dow is then defined as explained above in this sec-tion. Because of the reset function (see Section 3.1.2)applied to the data - which visualizes the position ofthe body-center at each BPbegin in the center of thewindow - the visualization gets completely alignedwith the user. The goal for the dancer is then to stayaligned in time with the displacement patterns visua-lized on the ground. If one succeeds, it means thatthe dance step was properly performed. This methodcould not yet be evaluated in a full setup. However,the concept of it provides promising means toinstruct dance figures.

3.4.2 Auditory monitoringThere have been designed ample computer technologiesthat facilitate automatic dance generation/synthesis from

Figure 9 An example of how to project the eight windows oneby one to create a real-time dance notation systemincorporating a first-person perspective.

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 10 of 16

music annotation/analysis [33-36]. The oppositeapproach, namely generating music by automatic danceanalysis, is explored in the domain of gesture-basedhuman-computer interaction [37-39] and music infor-mation retrieval [10]. We will follow this latter approachby integrating a dance-based music querying and retrie-val component in the Dance-the-Music. However, it isimportant to mention that this component is incorpo-rated not for the sake of music retrieval as such, butrather to provide an auditory feedback supporting danceinstruction. Particularly, the quality of the auditory feed-back gives the students an idea in real time how welltheir performance matches the corresponding teacher’smodel. As will be explained further, the quality of theauditory feedback is related to two questions: (1) Is thecorrect music retrieved corresponding to the dance fig-ure one performs? (2) What is the balance between themusic itself and the metronome supporting the timingof the performance?After a dance figure/music database has been created

(or an existing one imported) as explained in Section3.2, a dancer can retrieve a stored musical piece byexecuting repetitive sequences of the dance figure thatcorrelate with the basic step model stored in the data-base together with the musical piece. The computationalmethod to do this is outlined in Section 3.3.The procedure to follow in order to retrieve a specific

musical piece is as follows. The input buffer template isfilled from the moment the metronome - indicating thepredefined beat-pattern and tempo - is activated.Because the system needs the performance of one com-plete dance figure to fill the input buffer template (seeSection 3.3), the template matching operation is exe-cuted only from the moment the last sample of the firstcycle of the dance figure arrives. The number of themodel which is then indicated by the system as beingthe most similar to the input triggers the correspondingmusic in the database. To allow a short period of adap-tation, the “moment of decision” can be delayed untillthe end of the second or third cycle. The retrieval of thecorrect music matching a dance figure is only the firststep of the auditory feedback. Afterwards, while the

dancer keeps on performing the particular dance figure,the quality of the performance is scored by the system.The score is delivered by the correlation coefficient routputted by the system. On the one hand, the score isdisplayed visually by a moving slider that goes up anddown along with the r values. On the other hand, thescore is also monitored in an auditory way. Namely,according to the score, the balance between the volumeof the metronome and the music is altered. When r = 0,only the metronome is heard. In contrast, when the r =1, only the music is heard without the support of themetronome. The game-like, challenging character ismeant to motivate dance novices to perform the dancefigures as good as possible.A small test was conducted to evaluate the technical

design goals of this feature of the Dance-the-Music inan ecologically valid context. Moreover, it functioned asan overall pilot test for the evaluation experiment pre-sented in Section 4.For the test, we invited a professional dancer (female,

15 years of formal dance experience) to our lab wherethe OptiTrack motion capture system was installed. Shewas asked to perform four different dance figures in adifferent genre (tango, jazz, salsa and hip-hop) on fourcorresponding types of music played at a strict tempo of120 beats per minute (bpm). The figures consisted ofeight steps performed at a tempo of 60 steps per min-ute. The dancer was asked to perform each dance figurefive times consecutively. From this training data, fourmodels were trained as explained in Section 3.2 andstored in a database together with the correspondingmusic. Afterwards, the dancer was asked to retrieveeach of the four pieces of music one by one as explainedabove in this section. She performed each dance figuresix times consecutively. Because the dancer herself pro-vided the models, it was assumed that her performancesof the dance figures during the retrieval phase would bequite alike. The data outputted by the template match-ing algorithm (i.e., the model that most closely resem-bles the input and the corresponding r value) wasrecorded and can be seen in Figure 11. We only tookinto account the last five performed dance figures as the

Figure 10 r values when the model outputted correspondingly is similar to the intended model.

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 11 of 16

first one was needed to fill the input buffer. The analysisof the data shows that the model that was intended tobe retrieved was indeed always recognized as the modelmost closely resembling the input. The average of thecorresponding correlation values r over all performanceswas 0.59 (SD = 0.18). This value is totally dependent onthe quality of the performance of the dancer during theretrieval (i.e., recognition) phase in relation to her per-formance during the modeling phase. Afterwards wenoticed that smoothing the data contained in the modeland the data of the real-time input optimizes thedetected rate of similarity. As such, a Savitzky-Golaysmoothing filter (see Section 3.3) was integrated andused in the evaluation experiment presented in the fol-lowing section. Nonetheless, the results of this test showthat the technical aspects of the auditory monitoringpart perform to the design goals in an ecologically validcontext.

4 Evaluation of the educational purposeIn this section, we describe the setup and results of auser study conducted to evaluate if the Dance-the-Music system can help dance novices in learning thebasics of specific dance steps. The central hypothesis isthat students are able to learn the basics of dance stepsguided by the visual monitoring aid provided by theDance-the-Music application (see Section 2.2). A posi-tive outcome of this experiment would provide supportto implement the application in an educational context.A demonstration video containing fragments of the con-ducted experiment can be advised in a supplementaryfile attached to this article.

4.1 ParticipantsFor the user study, three dance teachers and eight dancenovices were invited to participate. The three teacherswere all female with an average age of 27.7 years (SD =1.5). One was skilled in jazz (11 years formal dance

experience, 3 years teaching experience), another insalsa (15 years formal dance experience, 5 years teachingexperience) and the last in samba dance (9 years formaldance experience of which 4 years of samba dance). Thesamba teacher had no real teaching experience but, dueto her many years of formal dance education, was foundcompetent by the authors to function as a teacher. Thegroup of students consisted of four males and fourfemales with an average age of 24.1 years (SD = 6.2).They declared not to have had any previous experiencewith the dance figures they had to perform during thetest.

4.2 StimuliThe stimuli used in the experiment were nine basicstep models produced by the three dance teachers (seeSection 4.3). Each teacher performed three dance fig-ures on a piece of music corresponding to their dancegenre (jazz, salsa, and samba). They were able to maketheir own choice of what dance figure to performwithin certain limits. We asked the teacher to choosedance figures consisting of eight individual steps andto perform them at a rate of 60 steps per minute (themusic had a strict tempo of 120 bpm). The nine basicstep models can be viewed in a supplementary fileattached to this article. They involve combinations of(1) displacement patterns of the feet relative to thebody-center, (2) displacement patterns of the body inabsolute space, and (3) rotation of the body around thevertical axis.

4.3 Experimental procedureThe experimental procedure is subdivided into threephases, following the three basic procedures of thedemonstration-performance method (see Section 2). Ademonstration video containing fragments of the evalua-tion experiment can be referenced in an additionalvideo file (Additional File 1).

Figure 11 A student interacting with the interface of the visual monitoring aid, projected on the ground.

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 12 of 16

Demonstration phaseIn the first phase, basic step models were inferred fromthe performances of the teachers. The three teacherswere invited to come one by one to the lab where themotion capture system was installed. First, the conceptof the Dance-the-Music was briefly explained to them.Then, they were equipped with IR-reflecting markers toenable us to use the motion capture system. After that,they were allowed to rehearse the dance figures on themusic we provided. When they said to be ready, theywere asked to perform each dance figure five times con-secutively. From these five cycles of training data, abasic step model was inferred. Each dance teacher wasasked to perform three dance figures, resulting in a totalof nine basic step models. Visualizations of these modelscan be referenced in an Additional File (2).Learning phaseWhat follows is a learning phase during which studentsare instructed how to perform the basic step modelsprovided by the teachers, only aided by the visual moni-toring system (see Section 3.4.1). As in the previousphase, the students were invited one by one to theexperimental lab. Also, they were informed about theconcept of the Dance-the-Music and the possibilities ofthe interface to control the visual monitoring aid, whichwas projected onto the floor (see Figure 2). After thisshort introduction, they were equipped with IR-reflect-ing markers. Then, the individual students were given15 min to learn a randomly assigned basic step model.During this 15 min learning phase, they could decidethemselves how to use the interface (body part selection,tempo selection, automated rotation adaptation, etc.).Evaluation phaseIn the last phase, it is evaluated how well the students’performances match the teachers’ models. All eight stu-dents were asked to perform the studied dance figurefive times consecutively. From these five cycles, the firstis not considered in the evaluation to allow adaptation.The performance is done without the assistance of thevisual monitoring aid. Movements were captured andpre-processed as explained in Section 3.1. The templatematching algorithm (see Section 3.3) was used to obtaina quantitative measure of the similarity (i.e., correlationcoefficient r) between the students’ performances andthe teachers’ models. Because an r value is outputted ateach BPbegin, we obtain in total 32 r values. The meanof these 32 values was calculated together with the stan-dard deviation to obtain an average score r for each stu-dent. Moreover, their performances were recorded onvideo in order that the teachers could evaluate after-wards the performed dance figures in a qualitative way.Also, after the experiment, students were asked to com-plete a short survey questioning their user experience.The questions concerned whether the students

experienced pleasure during the use of the visual moni-toring aid and whether they found the monitoring aidhelpful to improve their dance skills.

4.4 ResultsThe main results of the user study are displayed inTable 1. Concerning the average measure of similarity(r) between the students’ performances and the teachers’models, we observe a value of 0.69 (SD = 0.18). From aqualitative point of view, the average score given by theteachers to the students’ performances in relation totheir own performances is 0.79 (SD = 0.10). Concerningthe students’ responses to the question whether theyexperienced pleasure during the learning process, weobserve an average value of 4.13 (SD = 0.64) on a five-point Likert scale. The average score indicating the stu-dents’ opinion about the question whether the learningmethod helps to improve their dance skills resulted inan average value of 4.25 (SD = 0.46).

4.5 DiscussionFor the interpretation of the results, it is difficult to gen-eralize results in terms of statistically significancebecause of the relatively small number of participants(N = 8). Therefore, a more qualitative interpretation ofthe data seems more suitable. Although the samplenumber is relatively small, the average r of 0.69 (SD =0.18) suggests a considerable improvement of danceskills among all subjects due to the visual monitoringaid facilitated by the Dance-the-Music. Moreover, theaverage of the standard deviation of r (M = 0.06, SD =0.02) indicates that the individual performances of thestudents were quite consistent over time. These resultsare supported by the results of the scores teachers’gave–based on video-observation–to the students’ per-formances (M = 0.79, SD = 0.10). What is also of inter-est is the observation of a linear relationship (r = 0:50)between the scores provided by the template matchingalgorithm of the Dance-the-Music and the scores pro-vided by the teacher. Concerning the user experience,results suggest that students in general experience plea-sure using the visual monitoring aid (M = 4.13, SD =0.64). This is an important finding as the experience ofpleasure can stimulate students to practice with theDance-the-Music. Even more important is the findingthat the students in general have the impression thatthe Dance-the-Music is capable of helping them to learnthe basics of dance gestures (M = 4.25, SD = 0.46). Thissuggests that the Dance-the-Music can be an effectiveaid in music education.

5 General discussionThe results provided in Section 4 suggest that theDance-the-Music is effective in helping dance students

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 13 of 16

to learn basic step models provided by a dance teacher.Despite these promising results, some remarks need tobe made. First, the sample number (N = 8) was rela-tively small. This implies that, for the moment, theresults indicate only preliminary tendencies and can notbe generalized yet. Second, although the basic step mod-els involved combinations of displacement patterns ofthe feet and body and rotation around the vertical axis,the models were anyhow relatively easy. This was neces-sary because (1) the students had no earlier experiencewith the dance genre, and (2) because it was the firsttime they actually interacted with the visual monitoringaid (and we expect a learning curve for students to useand get used to the dynamic visual notation system).Therefore, in the future, it would be of interest to con-duct a longitudinal experiment investigating whether itbecomes possible to learn more complex dance patternswhen one becomes more familiar with the notation sys-tem presented by the visual monitoring aid. Third, inline with the previous remark, a comment made by theteachers was that dancing involves more than displace-ment patterns of the feet and body and rotation of thebody around the vertical axis. This is indeed a justifiedremark. However, as stated before (e.g., Section 1), theDance-the-Music can easily import other motion fea-tures and integrate them into the modeling logic basedon spatiotemporal motion templates. For example, inthe evaluation experiment, we integrated the horizontal,(x, y) displacement of the body in absolute space as asupplementary parameter in the model and visualizationaid. Apart from that, it must be stressed that the explicitintent of the Dance-the-Music–as it is presented in thisarticle–is to provide a platform which can help studentsto learn the basics of dance gestures which can then befurther refined by the dance teacher during danceclasses. Because the visually monitoring aid can in

principle also be used without a motion capture system,it can be useful for students to use the Dance-the-Musicto rehearse certain dance figures and small sequences ofdance figures at home before coming to dance class. Assuch, the time that teacher and students are togethercan be optimally spent without “losing” time teachingthe basics to the students.Technological realizations and innovations incorpo-

rated in the Dance-the-Music were developed explicitlyfrom a user-centered perspective, meaning that we tookinto account aspects of human perception and actionlearning. For example, the visualization strategy is basedon findings on the role of (1) segmentation of complexevents [15], and (2) a first-person perspective [16] forhuman perception and action learning. However, itmust be added that a third-person perspective (e.g., astudent watching the teacher performing) has its ownbenefits with respect to action learning [16]. Therefore,both perspectives must be considered as being comple-mentary to each other. Moreover, the introduction of anovel method for modeling and recognition based onspatiotemporal motion templates, in contrast to techni-ques based on HMM, facilitate to take into accounttime-space dependencies that are of crucial importancein dance performances. We also took into accountresearch findings stressing the importance of real-timefeedback of one’s performance [19,21-24]. Therefore, wedeveloped a recognition algorithm–based on templatematching techniques–that enables us to provide real-time, multimodal feedback of a student’s performance inrelation to a teacher’s model. Another property that wasessential in the design of the Dance-the-Music was thedynamic and user-configurable character. In essence, theDance-the-Music is considered as a technological frame-work of which the content depends completely on theuser (i.e., teacher and student). Users can propose their

Table 1 Descriptive overview of the results of (1) the quantitative (A) and qualitative (B) ratings of similarity betweenstudents’ performances and the corresponding teachers’ models, and (2) the user experience of the dance students(C)

Subjects 1 2 3 4 5 6 7 8 Average

Age 25 24 28 24 20 33 12 27

Model 2 7 8 1 6 9 4 3

Mean r 0.82 0.54 0.84 0.45 0.43 0.75 0.84 0.83 0.69 (SD = 0.18)

A

SD r 0.03 0.03 0.04 0.07 0.10 0.05 0.06 0.07 0.06 (SD = 0.02)

Teacher’s rating 0.9 0.8 0.9 0.6 0.8 0.85 0.8 0.7 0.79 (SD = 0.10)

B

0 = min, 1 = max

Pleasure 4 4 4 4 4 3 5 5 4.13 (SD = 0.64)

Educational potential 4 4 4 5 5 4 4 4 4.25 (SD = 0.46)

C

5-point Likert scale 1 = strongly disagree, 5 = strongly agree

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 14 of 16

own dance figures, music, tempo, etc. Moreover, theDance-the-Music facilitates to incorporate a broad spec-trum of movements (absolute displacement, rotation,etc.). These two features distinguish the Dance-the-Music from most dance games available on the com-mercial market that provide mostly a fixed, built-invocabulary of dance moves and music and also provideonly a small action space. Because of its dynamic char-acter, the Dance-the-Music can also have its benefits formotor rehabilitation purposes.

6 ConclusionIn this article, we presented a computational platform,called Dance-the-Music, that can be used in dance edu-cation to learn dance novices the basics of dance figures.The design of the application is considered explicitlyfrom a user-centered perspective, meaning that we tookinto account aspects of human perception and actionlearning. Aspects that are of crucial importance involve(1) time-space dependencies in dance performances, (2)the importance of segmentation processes and a first-person perspective for action learning, (3) the effective-ness of direct, multimodal feedback, and (4) the designof a dynamic framework of which the content is com-pletely dependent on the users’ needs and wishes. Tech-nologies have been presented to bring these conceptualapproaches into practice. Moreover, an evaluation studysuggested that the Dance-the-Music is effective in learn-ing the basics of dance figures to dance novices.

Additional material

Additional file 1: A demonstration video containing fragments of theevaluation experiment presented in Section 4.

Additional file 2: Visualization of the nine basic step modelsproposed by the three dance teachers participating in theevaluation experiment presented in Section 4.

AcknowledgementsThis study was carried out in context of the EmcoMetecca project fundedby the Flemish government. The authors want to thank Raven van Noordenfor her contribution to the three-dimensional visualizations, Ivan Schepersfor his technical assistance and Annelies Raes, Orphee Sergyssels and LeenDe Bruyn for their willingness to participate in the evaluation study.

Competing interestsThe authors declare that they have no competing interests.

Received: 15 April 2011 Accepted: 16 February 2012Published: 16 February 2012

References1. S Brown, M Martinez, L Parsons, The neural basis of human dance. Cerebral

Cortex. 16(8), 1157–1167 (2006)2. M Leman, Embodied Music Cognition and Mediation Technology, (MIT Press,

Cambridge, MA, USA, 2007)

3. M Leman, L Naveda, Basic gestures as spatiotemporal reference frames forrepetitive dance/music patterns in Samba and Charleston. Music Percept.28, 71–91 (2010). doi:10.1525/mp.2010.28.1.71

4. L Naveda, M Leman, The spatiotemporal representation of dance andmusic gestures using topological gesture analysis (TGA). Music Percept. 28,93–111 (2010). doi:10.1525/mp.2010.28.1.93

5. RI Godøy, M Leman, Musical Gestures: Sound, Movement, and Meaning,(Routledge, New York, NY, USA, 2010)

6. K Kahol, P Tripathi, S Panchanathan, Automated gesture segmentation fromdance sequences, in Proc 6th IEEE International Conference on AutomaticFace and Gesture Recognition (FG). volume = not speci_fied; pages = 883–888; publisher = IEEE Computer Society; location = Seoul, South Korea(2004)

7. A Ruiz, B Vachon, Three learning systems in the reconnaissance of basicmovements in contemporary dance, in Proc 5th IEEE World AutomationCongress (WAC), IEEE Computer society, Orlando, FL, USA, 13, 189–194(2002)

8. F Chenevière, S Boukir, B Vachon, Compression and recognition of spatio-temporal sequences from contemporary ballet. Int J Pattern Recogn ArtifIntell. 20(5), 727–745 (2006). doi:10.1142/S0218001406004880

9. K Kahol, K Tripathi, S Panchanathan, Documenting motion sequences witha personalized annotation system. IEEE Multimedia. 13, 37–45 (2006).doi:10.1109/MMUL.2006.5

10. F Bévilacqua, B Zamborlin, A Sypniewski, N Schnell, F Guédy, NRasamimanana, Continuous realtime gesture following and recognition, inGesture in Embodied Communication and Human-Computer Interaction, vol.5394. Springer Verlag, Berlin, Heidelberg, Germany, pp. 73–84 (2010)

11. C Bishop, Pattern recognition and machine learning, (Springer Science+Business Media LLC, New York, USA, 2009)

12. A Bobick, J Davis, The representation and recognition of action usingtemporal templates. IEEE Trans Pattern Anal Mach Intell. 23(3), 257–267(2001). doi:10.1109/34.910878

13. F Lv, R Nevatia, M Lee, 3D human action recognition using spatio-temporalmotion templates. Comput Vision Human-Comput Interact 120–130 (2005)

14. M Müller, T Röder, Motion templates for automatic classification andretrieval of motion capture data, in Proc ACM/Eurographics Symposium onComputer Animation (SCA), Eurographics Association, Vienna, Austria,137–146 (2006)

15. J Zacks, K Swallow, Event segmentation. Curr Direct Psychol Sci. 16(2),80–84 (2007). doi:10.1111/j.1467-8721.2007.00480.x

16. P Jackson, A Meltzoff, J Decety, Neural circuits involved in imitation andperspective-taking. Neuroimage. 31, 429–439 (2006). doi:10.1016/j.neuroimage.2005.11.026

17. D Davcev, V Trajkovic, S Kalajdziski, S Celakoski, Augmented realityenvironment for dance learning, in Proc IEEE International Conference onInformation Technology, Research and Education (ITRE), 189–193 (2003)

18. A Nakamura, S Tabata, T Ueda, S Kiyofuji, Y Kuno, Dance training systemwith active vibro-devices and a mobile image display, in Proc IEEEInternational Conference on Intelligent Robots and Systems (IROS), IEEEComputer Society, Alberta, Canada, 3075–3080 (2002)

19. J Chan, H Leung, J Tang, T Komura, A virtual reality dance training systemusing motion capture technology. IEEE Trans Learn Technol. 4(2), 187–195(2010)

20. L Deng, H Leung, N Gu, Y Yang, Real-time mocap dance recognition for aninteractive dancing game. Comput Animat Virt W. 22, 229–237 (2011).doi:10.1002/cav.397

21. D Hoppe, M Sadakata, P Desain, Development of real-time visual feedbackassistance in singing training: a review. J Comput Assist Learn. 22(4),308–316 (2006). doi:10.1111/j.1365-2729.2006.00178.x

22. E Gibbons, Feedback in the Dance Studio. J Phys Edu Recreat Dance. 75(7),1–6 (2004)

23. J Menickelli, The Effectiveness of Videotape Feedback in Sport: ExaminingCognitions in a Self-Controlled Learning Environment, PhD thesis, (WesternCarolina University, 2004)

24. S Hanrahan, R Mathews, Success in Salsa: students’ evaluation of the use ofself-reflection when learning to dance, in Proc of the Conference of TertiaryDance Council of Australia (TDCA), Melbourne, Australia, pp. 1–12 (2005)

25. K Kahol, P Tripathi, S Panchanathan, T Rikakis, Gesture segmentation incomplex motion sequences, in Proc IEEE International Conference on ImageProcessing (ICIP), IEEE Computer Society, Barcelona, Spain, 2, 105–108 (2003)

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 15 of 16

26. H Yang, A Park, S Lee, Gesture spotting and recognition for human-robotinteraction. IEEE Trans Robot. 23(2), 256–270 (2007)

27. T Artieres, S Marukatat, P Gallinari, Online handwritten shape recognitionusing segmental hidden markov models. IEEE Trans Pattern Anal MachIntell. 29(2), 20–217 (2007)

28. S Rajko, G Qian, T Ingalls, J James, Real-time gesture recognition withminimal training requirements and on-line learning, in Proc IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society,Minneapolis, USA, 1–8 (2007)

29. S Rajko, G Qian, HMM parameter reduction for practical gesture recognition,in Proc 8th IEEE International Conference on Automatic Face and GestureRecognition (FG), IEEE Computer Society. Amsterdam, The Netherlands, 1–6(2008)

30. T Hastie, R Tibshirani, J Friedman, J Franklin, The elements of statisticallearning: data mining, inference and prediction. Math Intelligencer. 27(2),83–85 (2005)

31. PJ Maes, M Leman, M Lesaffre, A model-based sonification system fordirectional movement behavior, in Proc 3th Interactive Sonification Workshop(ISon), KTH, Stockholm, Sweden, 91–94, (2010)

32. F Lv, R Nevatia, Recognition and segmentation of 3-d human action usinghmm and multiclass adaboost, in Proc 9th European Conference onComputer Vision (ECCV), vol. 3954. Springer Verlag, Graz Austria, pp. 359–372(2006)

33. H Mori, S Ohta, J Hoshino, Automatic dance generation from musicannotation, in Proc International Conference on Advances in ComputerEntertainment Technology (ACE), ACM Singapore, 352–353 (2004)

34. T Shiratori, A Nakazawa, K Ikeuchi, Dancing-to-Music Character Animation.Comput Graph Forum. 25(3), 449–458 (2006). doi:10.1111/j.1467-8659.2006.00964.x

35. J Kim, H Fouad, J Sibert, J Hahn, Perceptually motivated automatic dancemotion generation for music. Comput Animat Virt W. 20(2–3), 375–384(2009). doi:10.1002/cav.314

36. F Ofli, E Erzin, Y Yemez, A Tekalp, Multi-modal analysis of danceperformances for music-driven choreography synthesis, in Proc IEEEInternational Conference on Acoustics Speech and Signal Processing (ICASSP),IEEE Computer Society, Dallas, TX, USA, 2466–2469 (2010)

37. G Qian, F Guo, T Ingalls, L Olson, J James, T Rikakis, A gesture-drivenmultimodal interactive dance system, Proc IEEE International Conference onMultimedia and Expo (ICME), IEEE Computer Society, Taipei, Taiwan, 3,1579–1582 (2004)

38. G Castellano, R Bresin, A Camurri, G Volpe, User-centered control of audioand visual expressive feedback by full-body movements. Affect ComputIntell Interact. 4738, 501–510 (2007). doi:10.1007/978-3-540-74889-2_44

39. PJ Maes, M Leman, K Kochman, M Lesaffre, M Demey, The “One-PersonChoir": a multidisciplinary approach to the development of an embodiedhuman-computer interface. Comput Music J. 35(2), 1–15 (2011)

doi:10.1186/1687-6180-2012-35Cite this article as: Maes et al.: Dance-the-Music: an educationalplatform for the modeling, recognition and audiovisual monitoring ofdance steps using spatiotemporal motion templates. EURASIP Journal onAdvances in Signal Processing 2012 2012:35.

Submit your manuscript to a journal and benefi t from:

7 Convenient online submission

7 Rigorous peer review

7 Immediate publication on acceptance

7 Open access: articles freely available online

7 High visibility within the fi eld

7 Retaining the copyright to your article

Submit your next manuscript at 7 springeropen.com

Maes et al. EURASIP Journal on Advances in Signal Processing 2012, 2012:35http://asp.eurasipjournals.com/content/2012/1/35

Page 16 of 16