Shedding light on shadow: real-time interactive artworks based on cast shadows or silhouettes

Shedding Light On Shadow: Real-time interactive artworksbased on cast shadows or silhouettes

Christian JacqueminLIMSI-CNRS

& Univ. Paris-Sud91400 Orsay, France

[email protected]

Georges Gagner ’eDidascalie.net

1 rue Duvergier75019 Paris, France

[email protected]

Benoit LahozL’ange Carasuello

9 chemin du Trou Samson95150 Taverny, France

[email protected]

ABSTRACTDigital shadowing is a source of interest in immersive andinteractive artworks because it enhances the feeling of pres-ence, and because it is a very intuitive and engaging inter-action “device”. After a presentation of some major piecesin shadow-based artwork along two categories (silhouette-based digital shadows and cast shadow-based digital shad-ows), we propose a real-time software platform that can pro-duce these two types of shadows from video capture and im-age processing. The processing is divided into image anal-ysis for shadow extraction, calibration, and special effects.It is made highly flexible and parameterized, so that it canfit various configurations. Two specific configurations areillustrated because of their genericity: one for infrared lightsilhouette-based shadow and live performance, and one forboth digital shadowing categories and interactive art instal-lations with visible light. We then present some of the appli-cations of such a platform through vari ous live performancesmade by two theater companies involved in a collaborativeproject with scientists. The examples confirm the wide vari-ety of scenographic setups in which shadow can be involvedas a key component.

Categories and Subject DescriptorsH.5.1 [Information Systems]: Information Interfaces andPresentation—Multimedia Information Systems; Artificial,

Augmented, and Virtual Realities; J.5 [Arts and Human-ities]: Performing Arts (e.g. dance, music)

General TermsAlgorithms, Design, Experimentation, Human Factors

KeywordsAugmented Reality, Real-time Image Processing, Digital Art,Digital Shadow, Video-scenography, Interactive art

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.WOODSTOCK ’97 El Paso, Texas USACopyright 200X ACM X-XXXXX-XX-X/XX/XX ... $5.00.

1. INTRODUCTIONShadow is a recurring theme in arts, literature, and phi-

losophy since the Antiquity and Plato’s Cave where castshadows symbolize the limited perception of the world byits prisoners. Shadows in the visual arts have mainly ap-peared since the Renaissance through early perspective art-works. They were used to enhance the 3D perception of thescenes, and made light sources explicit. For this reason, per-spective and shadows can be considered as early attemptsto render artworks immersive by giving the user a feelingof “being there”. In addition to immersion and perspectiveperception, shadows were also used to give a sense of volumethrough shading. When combined with the real world, as inthe case of Masaccio’s fresco Tribute Money in Santa Mariadel Carmine (Florence), the light source of the artwork couldbe purposely chosen to coincide with a physical light sourcein the church, thus making this type of artwork an exam-ple of early Augmented Reality. Last, shadows have also beused for their symbolic meaning, representing evilness andhidden dark sides of humanity.

For similar purposes of immersion and augmentation ofthe real world, shadows can also be found in contemporaryvisual artworks, and we are mostly interested in interactivedigital arts in which video-projection reveals and highlightsshadows through digital shadows. Since this domain is madeup of a wide variety of works, and since the technological se-tups are quite diverse and rarely documented, our purpose inthis article is to offer a synthetic and cohesive artistic anal-ysis of recent digital works that refer to shadows, togetherwith an accurate description of our technological platformfor developing shadow-based artworks. To our knowledge,this is the first attempt to unify a body of works that haveplayed a significant role in the recent developments of digitalarts. This article intends to cover both the descriptive andanalytical artistic sides, and to address both the technolog-ical and scientific issues raised by that type of work.

In Section 2, we present interactive digital works that usevirtual shadow as a major component of the installation.These works are subdivided into two categories dependingon the input used to produce the virtual shadow, whethercast shadow or silhouette. Section 3 describes two combina-tions for the relative positioning of a video-projector, a cam-era, and a light source together with a platform developedfor the implementation of virtual shadow-based real-time in-teractive artworks. Last, in Section 4, several virtual shad-owing installations and performances based on the platformare described and analyzed together with their applicationsin social and educative activities.

2. DIGITAL SHADOW ARTWORKSInteractive artworks based on shadow offer an augmented

visualization of the physical world through the addition ofa digital shadow. Because of this configuration, all theseworks can be classified as Augmented Reality installationsin which the physical world is overlaid with digital informa-tion, here the video-projected augmented shadow. We definetwo categories of shadow-based artworks depending on thesource of the digital shadow: Cast Shadow-Based Shadow(CSBS) artworks for a physical shadow generally combinedwith video-projection, or Silhouette-Based Shadow (SBS)artworks for digital shadow built from a silhouette capture(see Figure 1).

Figure 1: Cast Shadow-Based Shadow (CSBS)through physical light occlusion, and Silhouette-Based Shadow (SBS) through silhouette projection.

The first category, CSBS artworks use a physical lightshadow as a basis of the visualization, and therefore relyon the extraction of the visible or non-visible shadow castby an occluder (the onlooker) with respect to a visible oran infrared (IR) light source. In these works the digitalshadow is produced from the capture of a cast shadow, andthe digital shadow must be accurately registered with thecast shadow it originates from. If the light source is in thevisible range, the overlaid digital shadow must fill accuratelythe dark area of the cast shadow with video-projected light.In the case of IR lighting, the digital shadow reveals theinvisible shadow through video-projection.

The second category, SBS artworks, is based on the ex-traction of the onlookers’ silhouette and its digital conver-sion into a virtual cast shadow with no corresponding physi-cal light source. Since the virtual shadow is video-projected,it is generally brighter than the background contrary to adigital shadow based on cast shadow that is projected intoa dark area. For SBS artworks, registration is less criticalthan for CSBS works since there is no physical shadow ithas to be aligned with. The criteria for alignment in SBSare mainly perceptual: the onlooker has to be convincedthat the shadow is really hers. The belief in the virtualshadow by actors does not depend solely on visual clues,but mainly results from the correspondence between theirown movements and the perceived shadow motions. At thevisual level, two main constraints exist however: the feetof the shadow should be located at the user’s feet, and theorientation of the shadow should correspond to the shadow

t hat would be cast if the camera was replaced by a lightsource. At the behavior level, the time delay between theshadow capture and the virtual shadow display should beshort enough to make the actor perceive that her own ges-tures are synchronized with the shadow movements. Wewill return to these issues in Section 4 when examining dig-ital shadow-based artworks. SBS and IR lit CSBS artworkscan be confused because of the invisibility of the physicalIR-light cast shadow in CSBS.

2.1 Cast Shadow-Based Shadow (CSBS) Art-works

Our interest in shadow-based artwork started when work-ing on video-scenography rehearsals for a theater play inwhich one of the spotlights was partially masked by a grid-shaped gobo1. Instead of considering video-projection andspot-lighting as two conflicting entities, we were exploringvarious mixes of these two types of light such as blending thevideo-projection with a dimmed colored spotlight. When us-ing the gobo, we came to the idea of video-projecting insidethe shadows of the gobo. For this purpose, we took a pictureof the partially shadowed stage from the video-projector lo-cation. Through a quick thresholding and calibration proce-dure, this picture was then converted into a black and whitebitmap and used to mask the video-projection in the lit partsof the stage so that it would only be visible in the shadows ofthe gobo. Maskin g and calibration were approximate, butthe inaccuracies of these processes instead of harming thevisual rendering, provided the spectators with a nice feelingof depth and volume. Later we decided to rework this setupin an interactive manner by analyzing a live video image,extracting automatically the dark zones, and using them asmasks for the video-projection. This is one of the purposesof the platform presented in Section 3.

Cast shadow has been used in several artworks to reveala “hidden” image that can only be perceived under specificconditions of lighting (a unique and accurately located pointsource) and shadowing (on a plane projection surface). Forexample, Tim Noble and Sue Webster2 build sculptures outof garbage or familiar objects to reveal images that tran-scend the reality of these objects by showing unexpectedshadow shapes such as a portrait or a sitting couple. Theseworks reverse the traditional meaning of shadow that repre-sents dark impulses or irrationality.

CSBS works draw their effects from the additive proper-ties of emitted light. In the case of a single light source,the light intensity in a shadowed zone is weaker than in theneighboring lit zone. For very powerful light sources, thedarkness of shadowed zones can be used to reveal video-projection that is too dimmed to be perceived in the litzone (e.g. Body movies by Lozano-Hemmer mentioned be-low). For the case of two or more light sources, CSBS workscan reveal video-images that are only visible if only one ofthe light sources is received. In Movie-in-Shadow [9], twovideo-projected images with complementary colors are accu-rately superimposed so that the resulting image is a uniformwhite color. By occluding one of the two light sources, thepassers-by reveal the complementary image inside their castshadows. A similar revelation process is offered by Rafael

1A gobo is a piece of metal with shaped holes that is placedin front of the spotlight and produces a pattern of light.2http://wikipedia.org/wiki/Tim Noble and Sue Webster

Lozano-Hemmer in Body Movies3 by showing video imagesinside the cast shadows of the passers-by. These shadowsare projected at various scales depending on the distance ofthe onlookers that are blocking the light of the spotlights.The variety in shape and scale of the silhouettes is combinedwith the video stream into a single fantastic scene.

Cast shadows can also be used in a playful way as a com-plementary graphical element to sculpture, whether staticor in motion. Kumiya Mashita4 and Fred Eerdekens5 bothuse a light spot that projects a single and clear cast shadowof a sculpture on a wall. The shadow reveals shapes of thesculpture that cannot be seen from the normal onlookersposition, and combines them with the visible shapes of thesculpture into a coherent and puzzling artwork. Closer tothe topic of our research, even though it has no digital pro-cess, the Night Shadow Project by Michael Neff6 consists inoutlining the shadows cast by public lighting at night. Ithas in common with our work to augment the cast shadow(graphically in his work and digitally in our approach), andto leave a trace of a light effect even when the conditionshave changed (through painting in his work, and through graphical feedback in ours).

All the CSBS artworks we have presented so far do notmake use of digital shadowing techniques, but rely insteadon a visible light shadow. On the contrary, the artworkspresented in the next sub-section (2.2) are based on digitalshadows obtained through the video capture and extractionof the silhouette. To our knowledge, these works representthe mainstream approach to digital shadowing even thoughit can also be based on cast shadow capture. The inter-esting feature of CSBS artworks is that they can result inthe complementary association of video-projection light withspotlights or natural lights. Our hybrid platform describedin Section 3 can process both CSBS and SBS setups.

2.2 Silhouette-Based Shadow (SBS) ArtworksShadow Play is an Asian traditional form of theater that

spread through Europe at the 18th Century and known inFrance as Ombres Chinoises (Chinese Shadows). Whereasoriginally designed for articulated flat puppets, Shadow Playhas since been extended to whole or partial human body an-imations. The basic setup consists in using a strong and hor-izontal light beam to project the silhouette of the puppets orperformers on a semi-opaque surface. This traditional formof theater has been retargeted to digital art by capturingthe video image of a silhouette and using it as a mask forsynthesizing a virtual shadow through video-projection.

The SBS digital artworks can be roughly divided into twocategories depending on whether the virtual shadow is usedas an interactor or as a visual and aesthetic component in theinstallation. The reason for using a silhouette-based shadowas an interactor, is to “simplify” the onlooker’s presencein the installation by turning her into a legible and simpleblack shape. The silhouette keeps enough similarity with thebody to make interaction transparent and intuitive, while itavoids clutterring up the visual rendering of the work with arealistic view of the user. In such installations, it is generallythe border of the silhouette that serves as an interactor.

3http://www.lozano-hemmer.com/body movies.php4http://kumiyamashita.com5http://www.fred-eerdekens.be/6http://michaelneff.com/

A pioneering use of the SBS setup is the Videoplace en-vironment created by Myron Krueger in 1977. Originallydesigned as a responsive environment, this artwork devel-oped “the basis for a new aesthetic medium based on real-time interaction between men and machines” [6]. The per-former invited to play in Videoplace was interacting with thecomputer-generated images through her SBS shadow. Theinnovation in Videoplace was to consider interacting throughthe user’s shadow as the main artistic issue in this environ-ment, upstaging the visual rendering accordingly. Despitesuch early shadow-based approaches to interaction, thereremains numerous possibilities and future exciting artworksthat can certainly further this line of artistic research, suchas David Rokeby’s Silicon Remembers Carbon7.

Some of the most representative examples of SBS instal-lations are Zachary Booth Simpson’s Mine-Control8 works.In his works, the silhouette contour is used as an obstacleto control a flow, the bouncing of a ball, the motion of ananimal and the waves on a surface. In his article [8], GolanLevin quotes several digital artworks controlled by hand andarm shadows, including his own work Messa di Vocce9 (incollaboration with Zach Lieberman, Jaap Blonk, and JoanLa Barbara), in which silhouette is used to generate visualelements combined with voice analysis.

The SBS artworks that exploit shadow as an aestheticcomponent generally do not assign a specific interactive func-tion to the user’s shadow projection. The shadow can betaken as it is, as a schematic and yet recognizable trace ofa presence. Such SBS digital artworks are the ones closestto the ancestral form of shadow play. Scott Snibbe’s Out-

ward Mosaic10 collects onlookers’ animated shadows into alive and refined gallery of multi-scale loop videos of shad-ows. The control relationship between the user and hershadow is reversed when the shadow appearance is modi-fied by the user’s behavior. In Philip Worthington’s ShadowMonsters11 hand shadows are turned into monster’s headprofiles through the addition of frightening graphical ele-ments.

Live performance has also a strong interest in introducingshadows on stage for various scenographic purposes. For ex-ample in Philippe Decoufle’s Sombrero12 ballet, shadow isused for scale effects or mirror behaviors. A wide range ofshadow-based scenery effects are also presented in ChunkyMove’s Mortal Engine dance performance. These effectscombine music- and gesture-controlled generative graphicswith silhouette capture to immerse the dancer’s shadow intoa flow of hypnotic visual effects.

By altering the behavior of the shadow and introducingdiscrepancies between the shadow appearance and the hu-man body that produces it, the pseudo-shadow is likely toquestion our presence to the world. Zachary Booth Simp-son’s Shadow13 with Adam Frank presents a virtual shadowthat has no physical cause, and that reacts to and combineswith the onlooker’s own cast shadows. Similarly, Digital

7http://homepage.mac.com/davidrokeby/src.html8http://www.mine-control.com/9http://www.flong.com/projects/messa/

10http://www.snibbe.com/projects/interactive/11http://www.worthersoriginal.com/viki/12http://www.culturekiosque.com/dance/-reviews/decoufle sombrero.html

13http://www.mine-control.com/shadow.html

Shadow14 by Alessandro Corsini and colleagues introducesso-called rebellious shadows that quit their mirror behaviorto switch to unexpected motions corresponding to previousrecordings of the same silhouette. We will present similar in-vestigations in Section 4 for live performance with children,teen-agers, and young disabled adults.

Last, shadow can also be used in artworks as a mediumbetween the real world and a virtual one. In the Abstract15

installation by Joelle Bitton, visitors are invited to a medi-tation Japanese garden world by discovering it through theirtransparent shadows. In the Augmented Virtuality instal-lation Gate 3.5 16 by Bertrand Planes, the onlookers havetheir shadow projected inside a 3D virtual world. In ad-dition to enhancing their feeling of presence in the virtualworld, the shadow also serves the purpose of a kind of “non-tactile” contact with the virtual world, and a discovery ofits geometry through virtual shadow-based exploration.

3. A PLATFORM FOR SHADOW-BASED IN-STALLATIONS

This section describes a generic platform for shadow and/orsilhouette extraction, and its use for various output effectsthrough image synthesis. The input to this software is an IRmonochromatic image that possibly contains both a silhou-ette and its cast shadow. The output is one or two virtualshadows rendered from one or both of them.

The setup for experimenting with shadows is presented inFigure 2. It shows the most complex case with two typesof cast shadows: the blocking of the video-projector lightand the blocking of the spotlight (visible or IR light). Thevideo-projector light shadowing is considered as an uninter-esting parasite shadow. We therefore focus only on the castshadow that results from blocking the light emitted by thelight-spot. In the case of visible light, the digital shadowmust be calibrated to the physical shadow. The setup fordigital shadowing offers a wide variety of combinations thathave been studied exhaustively by Chan and Gagnere [4].They mostly depend on the relative positions of the user, thecamera, the video-projector, the spotlight, and the “screen”.We only focus on two emblematic and illustrative ones; theyare illustrated by Figures 3 and 4.

First, Figure 3 shows a top view of a configuration dedi-cated to SBS that has been investigated by the theater com-panies involved in the project and in this article. In thisconfiguration, a translucent screen is located behind the per-former and receives both the video-projection and a power-ful IR light so that the performer’s silhouette can be easilyextracted from the lit background. Rear video-projectionavoids the performer’s occlusion of the video-projector lightthat otherwise can occur in front projection. This configura-tion is well-suited for live performance because the shadowis clearly visible for the audience, and does not require com-plex calibration procedures. It has been used in several per-formances and educational activities with children by thetheater companies. They are presented in Section 4.

Second, the configuration shown in Figure 4 has only beenexploited in the laboratory work for tuning up the platform.This second configuration relies on image analysis for the

14http://www.digitale-medien-bremen.de/15http://www.superficiel.org/joelle/art/pages/abstract.htm16http://www.bertrandplanes.com/-pages/Gate35%20Panorama.php

Figure 2: Image input/output in the platform forshadow-based works.

extraction of physical cast shadow.

Figure 3: Top view of SBS IR light configuration,and sample performance.

The two main steps in computational processing for digitalshadow-based installations are image analysis for real-timeshadow or silhouette detection, and visual effects throughimage synthesis (see Figure 5). Both are implemented withGPU-programming (shaders) for optimization and portabil-ity purposes.

The image analysis modules extract two masks from aninput B&W video image, corresponding respectively to castshadow and silhouette by distinguishing the pixels accord-ing to their homogeneity: shadows tend to be more homo-geneous than silhouettes. The image analysis modules arefollowed by an image synthesis component that uses theshadow and silhouette masks to render a virtual shadowfrom one of these elements or both, and adds optional artis-tic effects such as fire, water, particles, virtual 3D compo-nents, etc. In the case of cast shadow input (CSBS works),the output shadow is calibrated so as to be embedded intothe cast shadow. In the case of silhouette input (SBS works),the virtual shadow is generally displayed as a fake cast shadow,anchored at the feet of the onlooker, and projected oppo-site to the camera considered as a virtual light source. Weexamine these two components (image analysis and imagesynthesis) in turn.

Figure 4: Top view of IR or visible light CSBS andSBS, and sample laboratory capture.

Figure 5: Full pipe-line.

3.1 Real-time Image AnalysisComputer Vision plays a central role in Augmented Real-

ity applications because it is used for registering the virtualelements with the physical world through 3D reconstruc-tion and calibration procedures. Since the augmentation ofshadow in CSBS and SBS artworks is based on the video-projection of a digital shadow, these works belong to SpatialAugmented Reality (SAR) described by Bimber et al. [2]. InSAR, video-projection is used to overlay digital informationon the physical world by carefully registering the digital in-formation with its corresponding location inside the physicalworld.

In CSBS works, the cast shadow of the user blocking alight beam (whether visible or IR) must be extracted. InSBS works, the user’s silhouette has to be obtained andprojected from a virtual point light into the digital scene.In both cases, cast shadow and silhouette detection are madein the IR spectrum to avoid visual feedback by capturing thevideo-projected image by the video camera (video-projectorlight contains very little IR and is almost invisible to an IRvideo capture). The IR lighting is obtained either by usingan IR spotlight if the installation is in the dark, or by usinga visible light spotlight that contains IR in the case of CSBSworks. In both cases, video capture for image processing ismade solely in the monochromatic IR spectrum.

Whereas IR image analysis has been extensively studiedto track objects that emit IR light, the issue in our workis to track parts of the image that do not receive IR light:either cast shadows due to the user’s occlusion (Figure 4),or silhouette of the user(s) standing in front of a bright sur-face or a translucent surface on which IR light is projected

(Figure 3). The image processing part of SBS works is asimplified form of chroma keying in which a silhouette isseparated from its background. Instead of color separation,silhouette extraction relies here on light intensity separa-tion and background subtraction because it is made on amonochromatic IR image. Similarly, cast shadow extractionis also based on the detection of darker areas in the image.The modules for image processing are given in Figure 6.

Figure 6: Image analysis pipe-line.

After video deinterlacing, the first part of image process-ing concerns image calibration. First, the deinterlaced IRimage is transformed into a plane image through lens radialcorrection. Then two transformations are applied (rotationand perspective) in order to register the image to the user’sposition in the scene. After calibration the video-projectionof the cast shadow matches the physical cast shadow in caseof a CSBS installation.

The second step in image processing consists in cast shadowand silhouette extraction. First a background image is com-puted through a median and Gaussian filter on 5 imagescaptured over 70 frames [7]. The background contours arefiltered and combined with the background image for shadowand silhouette extraction through background subtraction.Contours are highlighted in the extraction process becausesome of the renderings use them either for aesthetic purposes(outlining the shadow edge) or for functional purposes (us-ing the contour of the shadow for interaction). Backgroundsubtraction is based on logarithmic intensities because itresults in better quality, and is less sensitive to thresholdselection [11]. A final filter separates the cast shadow fromthe silhouette by using the property of background subtrac-tion resulting in more homogeneous regions in the shadowedpart of the scene than in its mobile parts [7].

3.2 Real-time Image SynthesisThe masks associated with shadow and silhouette are used

by the image synthesis modules to generate virtual shadows.Three main components build up the image synthesis partof the platform: a module for computing and highlightingthe bounding boxes around the two masks, a particle engine,and the rendering modules for visual effects.

The tracking module relies mainly on a geometry shaderthat computes the minimal and maximal coordinates of the

shadow and silhouette areas to obtain and display theirbounding boxes. These data can be used, for example, bythe particle engine to attract particles towards the center ofthe bounding boxes, and make them converge towards theshadow or the silhouette whatever their positions in the fieldof view of the camera. The bounding box is calculated on asub-sampled image to reduce the computational cost.

The particle engine belongs to the category of physicsengines. They are very popular in the game and digital artcommunities where they are used to add realism to anima-tion. Particles are point masses that are controlled by a forceand interactions with other particles or with obstacles. Wehave developed an open GPU framework for animating andrendering particles. Fragment shaders compute the particledynamics and vertex shaders update the positions throughvertex texture fetch. The engine allows for the computationof particles bouncing on the contour of an object (here theshadow or the silhouette) according to the surface norm.

Two artistic particle renderings have been studied. First,an ambient visual display shows falling particles as whitedrops, and lets them bounce on the surface of the shadows.Second an interactive rendering shows particles as wordsthat are attracted by the center of the shadow, and bounceon its surface. If the shadow remains still for a while, thewords will eventually stabilize on its surface.

The last processing pass of virtual shadow rendering con-sists of a wide variety of fragment shaders that take as inputthe calibrated and distinguished areas of shadow and silhou-ette, and use them to generate various visual renderings ofa digital shadow that are projected in the physical shadowin the case of CSBS artworks, or anchored to the user’s feetand projected on a plane surface (wall, vertical screen, orfloor) for SBS works. We only focus on a few renderingmodules that were of particular interest for the technicaldevelopment of artistic installations.

The purpose of the colorimetric compensation [1] mod-ule is to “erase” the shadow by video-projecting a brightvirtual shadow that compensates the decrease of illumina-tion due to cast shadow. This module is targeted for CSBSinstallations with a visible light. Since no full visible lightrange video capture is available in our setup, the compensa-tion cannot rely on a captured image and on the comparisonof the lit and non-lit areas. It is based on the manual tun-ing of a uniform lighting of the floor surface. Better resultscould be obtained through live color video capture.

The feedback module is mainly intended for installa-tions with non-visible light. Its purpose is to reveal a shadow(whether CSBS or SBS) and to display it with a feedbackrendering loop in order make the trace of the user’s motionexplicit. This module can be applied to various video fillingsof the digital shadow: uniform color, texture, or video. Thiseffect also gives the onlookers a clue to the user’s motionspeed according to the length of the shadow feedback trail.

In the same vein as the particle engine, two modules forfire and a water effects modify the shadow masks at thepixel level, based on a simple physics of fire or water, and anoise texture to add randomness to the motion and the col-oring. In the fire module, the shadow is colored with a mov-ing and distorted fire texture, and the border pixels are dis-placed towards the top of the image in combination with thenoise texture. The geometric distortion and color changesof the water effects are controlled by particle displacementsgenerated by the particle module described above.

Last an Augmented Reality module is the combinationof a texture or a fixed video and a mask that highlightsa foreground element of the image on which the shadowcan be cast. This kind of simple augmentation can onlywork for an image of a plane shadowed surface such as afence or a wall. For more complex geometrical scenes, it isnecessary to reconstruct the geometry of the photo or videoas described by Jacquemin, et al. [5]. Another solution is touse the shadow for Augmented Virtuality by projecting theflat shadow inside a 3D virtual scene, in which the geometrycan be used to compute the shadowed areas.

4. SAMPLE SHADOW ARTWORKSAs an application of the platform described in the pre-

ceding section, this section focuses on examples of SBS art-works, and offers new insights to digital shadowing. Thereare natural motivations for using SBS in the performing arts:The tradition of Shadow Play makes the audience conven-tionally understand the SBS as a shadow projection of theactor and dancer due to a virtual horizontal light placedin the audience. The lack of continuity between the per-former and her shadow does not break this rule. On thecontrary, it reinforces the magical effect of projection thatcan be found in many traditional narratives such as Plato’sCave or Pliny’s Clay modeler Butades. Last, using a shadowcast on a floor is not easily perceived by an audience becausethe perspective dramatically diminishes the visual effect ofhorizontal images, and because a single spotlight makes itdifficult to light up the whole stage.

4.1 Sample SBS Dance ScenographySBS has played a major role in the theater play Les Reve-

lations d’une ombre17 by co-authorGeorges Gagnere in whichthe seemingly autonomous shadow of a dancer played in timeand space with the performer. Four real-time effects wereused

• In Figure 7.a, the SBS setup allows us to producein real-time chronophotographic views of the dancer’smovements, inspired by the famous 19th century Muy-bridge’s studies on human locomotion. The variousshadows are triggered by notes of music played onstage by a saxophonist.

• Again through the SBS setup, the live shadow of thedancer is shrunk and put in a virtual maze as if shewere located inside a video-game (see Figure 7.b). Thedancer moves her body and finds the appropriate in-teractions that will make her progress.

• Figure 7.c shows the cast shadow of the dancer in thewhite video-projected light behind a screen. The sil-houette is shot by a camera in front of the stage andis symmetrically projected as a digital shadow on thesecond half of the screen. This setup allows us to cre-ate several doubles with delays and buffered imagesthat multiply the possibilities of additional characters.

• Last, with the SBS setup and a symmetry of the pro-jection, the dancer plays with her shadow to discoverstrange topologies musically created in real-time by asaxophonist (see Figure 7.d).

17http://www.didascalie.net/lrdo

Figure 7: Les Revelations d’une ombre, 2009.

The use of the setup presented in Figure 3 offers appro-priate conditions to allow the SBS be out of the performer’scontrol. It opens the possibilities to experiment a dialoguein real-time with the living shadow projection of the dancer(among others, inspiring autonomous shadow examples arethe Peter Schlemihl’s Remarkable Story by Adelbert vonChamisso, or The Shadow by Hans Christian Andersen).The theme of the monstrous metamorphosis was developedfor expressing the secrete ambitions of control from the mu-sician character towards his dancing creature. The setupoffered possibilities to develop an initiatic quest by havingthe shadow counterpart making an extraordinary journey tofree the obsessed spirit of her creator.

4.2 Sample SBS Theater ScenographyThe purpose of L’ange Carasuelo company SBS work is to

make possible a shared experience between scientists, artists,technicians, and audience. One of its goals is to propose ashared and simple “shadow tool” for artistic and educativepurposes, considering that many performance artists, tech-nicians, and citizens find digital technologies challenging ascreative tools whereas they are commonly used for enter-tainment (e.g. video-games).

This work in strongly related with the deep meaning of theshadow, as a connection between us, our “invisible world”,and the real world. Based on this assumption, the researchstarted from a corpus of texts and representations from CarlJung’s work, recent digital shadow works mentioned in Sec-tion 2, and Haruki Murakami’s novels. This ternary rela-tionship had important consequences on the use of the plat-form, both for artistic and mediation approaches.

Firstly, writing with (and somehow, for) the tool implieda shift in the roles of each member in the artistic team.Artists had to be able to deal with technical constraintsfor performance design, and technicians had to understandthe artistic needs, and propose “on the fly” some simple,fast, and efficient digital environments. Secondly, shadowwas used to inform the audience and the performer of thespace and time structure of the virtual world, by shiftingour point of view on the world, offering an open gate to ourown imagination. As detailed in the next section, the digitalshadow played the role of an avatar that could enlarge theconsequences of our motions in an immersive context. Thispoint was particularly meaningful during the workshops weorganized at the ESSEC Business School, hosting physically

and sensorially handicapped students.

4.2.1 Technical constraints and believabilityThe first appropriation of the platform was to organize

workshops, brain-storming, and discussions with the part-ners. Thanks to the artistic collaborative company dida-

calie.net and to the efforts made jointly by the artistic andtechnical participants, we designed Tolalusulo18, a workableversion of the tool that could be used to experiment variousscenographic setups based on different configurations of thetracking and projection surfaces. We finally chose a frontalinstallation such as the one described in Figure 3 integrat-ing: an IR translucent screen for vertical projection, a shinyfloor to reflect the IR light behind the feet so that the IRcamera in front the screen could catch the whole silhouetteof the performers, a video-projector, and a set of theatrespotlights equipped with visible light filters installed behindthe screen.

The main trouble we had to face was that our setup wasshrinking the space in which the participants could per-form. Restricted to one meter away from the screen, thisnarrow stage band was constraining the actor to a kind of2.5-dimensional space. As a consequence we decided to writethe show with this constraint: our characters would be en-closed in his space, and their shadows would deploy in a 3Dspace behind the screen. A kind of unconscious personalspace was suddenly revealed by their independent shadows.In this way, the 2.5D performing space was expanded to amuch larger world (3D space plus time, with the possibilityof going back in time) through 2D shadows. Since we couldnot use traditional spotlights on the performers, we decidedto keep the performers in the dark and to swap the roles:darken the actor, while lighting up her (digital) shadow.

These constraints, combined with a time delay throughsoftware recording capabilities, became the main ingredi-ents of a story we wrote on stage. A text was written byBenoıt Lahoz based on Haruki Murakami’s novels, and theopening nights took place in L’apostrophe, Cergy and Le

Cube, Issy-les-Moulineaux during Winter 2011 (Figure 8.e).Working with the platform impacted significantly our writ-ing practices, and oriented us toward a layer-based com-positing style. The story turned into a journey into layersof memories and social data received by the characters, inwhich the shadow was a kind of guide and a support forletting them reveal their deep personal values.

SBS was also used as an interactor with virtual objectslike a 3D moving wall, a 2D fluid wallpaper, etc. Both theshadow as a self-representation and as an interactor weretightly coupled to make it understandable by the audience.In some scenes, the shadow was projected onto a 3D virtualworld used as a stage background. Opening a 3D world tothe shadow had to be believable for both the performer andthe audience so that it could be perceived as a projectioninto a fragile Augmented Virtuality environment.

Believability was also a major issue in the experimentationworkshops with pupils, students, and individuals (January2010-June 2011). Early 2010, the major negative commenton SBS setup was about latency. The 6-year old pupils werethe most concerned by this issue. In 2011 we reached ap-proximately 30 frames per second and reduced so much thelatency that it became almost imperceptible, although still

18http://www.didascalie.net/tolalusulo

Figure 8: Taverny workshops, 2010, and Un petit a

cote du monde play, 2011.

noticed by the performers. This persistent observation un-locked another interpretation of the participants: What isactually important is not a merge between our cast shadowand the projected shadow, but the possibility for a performerto identify herself with her own shadow. As soon as we canidentify ourselves with the projected black shape, all possi-ble distortions can be applied without cancelling this per-sonal identification. This consideration opened a very largeway for dramatic experimentations: What is happening, inwhat kind of world, between the moment I move and themoment my shadow moves?

4.2.2 Shadow deconstructionThe first intention for the performance was to open a vi-

sual space-time to be experimented with on stage, both bythe characters and the audience. It was based on shadowspatial deconstruction by casting it on diversely oriented vol-umes, and on temporal deconstruction through latency ren-dering that would result in time-inconsistent shadow. Ourhypothesis was: as the character (then the audience) focuseson her shadow, she provisionally breaks her so-called unityto enter a deeper space-time of dreams, memories, and infor-mation structures. This assumption is related to the conceptof theater that opens a space-time window inside the audi-ence’s own chronology, in which the spectator explores herown personality through identification and catharsis. Ourresearch focused on Carl Jung’s work on shadow, “collectiveunconscious” and “metaphysical reality”.

By dealing with technology we played with four dimen-sions (3D space + time) to enter this particular world. Thetwo main tools were: mapping the SBS on a 3D based vir-tual space, and increasing or decreasing the latency whileperforming. Thus the performer was informed of her pres-ence in this world through her shadow, by travelling a spacelarger than the stage (corresponding to a specific mentalspace) in a deconstructed time: delaying the past while al-lowing the audience to catch a glimpse of the future.

The complex relationship between light source, projectionsurface, and obstacle made the shadow take a difficult to

grasp piecewise pass (like when our shadow is cast on per-pendicular surfaces and broken into scattered pieces). Con-sequently, the shadow cast on the “walls” of this virtualspace had to be made in a realistic manner to look believ-able. As the scenery had to be portable, we used a proce-dural process to map the shadow image on a virtual space(as described by Jacquemin et al. [5]). These constraints re-vealed a parallel between the technological problems we hadto resolve and the inner questions of the performance similarto the issues raised by Sheppard et al. in [10] on the waya dance performance can be staged in a very technologicaltelepresence environment.

Our deconstructed character, receiving information fromtoo many sources, was about to travel in a virtual world de-constructed in space and time. Layers of information wererepresented by vertically and horizontally controllable per-pendicular surfaces on which the cast shadow revealed pre-recorded pictures, videos, and structures (especially Rutt/-Etra-like heightfields), and then zoomed to other spaces.

By opening virtual spaces, we took the risk to make theperformer disappear by not being strong enough to inhabitand “dominate” the virtual environment. To do so theperformer had to enter in a genuine relationship with theshadow: she had to keep in mind what she made one secondago and then react by inventing another continuity whileperforming inside the moving space. Through this processthe technology became a tool for improvisation: the per-former, with the help and creativity of the technicians play-ing live, was able to inform herself of what she was doing andof the consequences of these acts in an oscillating time. Ul-timately immobility suddenly closed the spaces and stoppedthe time, in a kind of rhythmic ritual experience, not so farof the origins of theater.

4.2.3 Shadow playsBased on these issues, we built a few games for workshops

with children and teen-agers:

• The Shadow Phone in which 4 participants were play-ing the “mirror game” with their own shadow beforewe shuffled the shadow videos so they could play livewith someone else’s shadow (Figures 8.b & c).

• The Shadow Ballet in which 6 video-captures of danc-ing children were buffered and projected as shadowsilhouettes with various delays in different colors on apicture or video background. It ended up into a dance

floor of the past moment in which each child tried torecognize the shadows’ owners (Figure 8.d).

• InWhat is inside my shadow, the children were promptedto use their shadow to reveal a picture or video theyhad chosen, and comment on it aloud while making itvisible.

• My image that belongs to us proposed to the partici-pants to create a collective image with their own shadowin a sequential manner, each of them leaving a traceof their presence on the screen.

• Shadow Pong was an adaptation of the well-knownPong game that could be played by 2 participants us-ing their shadow as interactor.

Other games were implemented with motion freezing, hor-izontal shadow flipping, buffering, inverting colors, etc., al-ways associated with a debriefing and observation time.

The imagination and creative skills of the participantswere practiced with their whole body, so that the interfacewith the virtual world was not reduced to a limited devicesuch as a joystick or a Wiimote. The whole body was usedto make the members of the audience understand the con-tinuity between us, our shadow, and the virtual world, andmake them feel projected into their own “shadow world”(imagination, unconsciousness, etc.).

Working with people who have physical, linguistic, or per-ceptive disabilities, far from limiting the possibilities multi-plied them. For example, a partially-blind student passedlong hours drawing the background of a platform game, andthen used the shadow as an avatar. In this way his wholebody (and not only his eyes or fingers) was involved into thegame. Time latency was staged with students suffering fromdyspraxia who were actually experiencing a latency betweenstimuli reception and processing. This correspondence be-tween the real human and the virtual shadow resulted ininteresting discussions with the participants and offered ussome drama material for the theater play we were buildingtogether: What is happening during this transitional time?Is the time of imagination, fiction, and poetry?

Perceptive amplification was the main issue in this collec-tive experience. In sharing this tool, we acquired knowledgeand competences that led our artistic research to new un-explored paths that we could never have found just on ourown. This was made possible thanks to the partnership be-tween scientists, artists, education and civil society staff,and individuals.

4.3 Art/science collaboration processThe realization of the software platform and its use for

performances and pedagogical workshops shows how scien-tific and technological works intertwine with artistic creationand offer new perspectives for both disciplines.

The first intuition shared by scientists, engineers and art-ists in their collaborative project was to use the shadow tobuild a bridge between real and virtual worlds. We alreadysaid that by covering a space the shadow gives a proof ofpresence. This is not easy to fully understand (children doit only around five years old), but it is playful to experiment.Playing with the shadow of a performer or an onlooker re-sults in new interactions setup between the real and virtualworlds. Starting from a digital shadow, we imagined con-crete artistic situations to physically interact with digitalaugmentation of the physical world.

The first point was that building a digital shadow wasa very difficult step either for scientists and engineers orfor artists and scene directors. The use of visible and in-visible lights, video-projector, camera, computer, and pro-jection surfaces, the spatial relationship between the per-former/onlooker and her shadow resulted in a complex ex-perimental environment. Finally, the scientific methodologywas of a great help to decompose the complexity and tochoose the appropriate setups. An amazing result was thatthe computer program does not grasp the concept of shadow.What a child’s brain can do in one second is unachievable bythe most powerful computer. Consequently the whole staff(scientific, technical or artistic) had to make common choiceto build the digital shadow and to accept limitations in its

use: we shared a common object, built it from scratch, andlearned together, step by step, how to use it in an augmentedenvironment.

The art/science collaboration process refutes the notion ofprogress and implements well a post-modernist approach tolife in which errors and bugs, practices and habits, mean-ing and concepts count more than technological achieve-ments and scientific advances. As a consequence, such acollaboration can only mature if both parts collaborate inan empathic and listening manner, and accept to dismisstheir preconceived views on technological art. Such a needis well illustrated by Sheppard et al. in [10] in the TED tele-immersive dance environment experience, in which bugs, de-lays, and discrepancies are used as artistic tools, shiftingfrom an approach where they should be avoided to a set-upin which they are integrated and controlled by the dancers.

At the beginning of our collaboration, performers playedwith the digital shadow as an additional character, and ex-plored its autonomy. They imagined scenes that inducedspecific scientific or technological explorations to make pos-sible interactions. Conversely, the scientists and engineersexplored “by themselves” interesting properties of the digitalshadows that had aesthetic qualities, and brought originalset-ups to artists. For instance, tracking and particle effectswere suggested by artists, but water and fire effects camefrom an scientific exploration of the particle engine appliedto distort the digital shadow. To have built together a com-mon object allowed independent creativity from each partic-ipant and genuine investigation producing relevant results ineach field (scientific publication or shows), and finally helpedto reach new territories of collaboration.

This process took several months before being mature. Itwas not planned when the collaboration started. We reg-ularly used to express the need of a common vocabularyto understand the multiple and seemingly divergent emerg-ing points of view. But the success in collaboration camefrom fully sharing a common object and its setups, a digitalshadow interacting with a virtual world in an augmentedenvironment. As we wrote in part 2, there are two mainsetups for dealing with shadows: CSBS and SBS. We justexplained why we first explored the SBS. Nevertheless, cur-rent developments are actually led by the scientific partnerwho convinced his artistic partners to explore the CSBS bydeveloping the potentialities of the little more tamed digitalshadow. The initial exploration of the autonomous shadow,and the consequent common experience, can apply now toother contexts in which a cast-shadow can be transformedfrom inside. New artistic ideas will come from forthcomingworkshops inv olving light designers, stage directors, and sci-entists. We hope to produce some works that will developthe digital part of the CSBS framework.

In our project, the shadow theme took a shape that al-lowed to produce in the same time scientific knowledge andartistic experiments. We were certainly helped by the origi-nal double nature of the shadow: it played a important partin the discovery of our world (principally in astronomy andin cognition as shown by Casati [3]), and it was for all time afavored mean to tell stories. Using it to explore AugmentedReality gave us the opportunity to find out new topics ofresearch and practice.

4.4 Artistic dissemination and sustainabilityWorking with technologies inside a collective project made

us question the economic aspects of project: How could weshare the results and the tool when the project is finished?We had to find affordable solutions to install a basic softwaresuite and consider the best low-cost “tool quality-sharing op-portunities”. Incidentally or not, these three aspects of ourapproach (technical, psychological, and social) were alwaystightly linked to our artistic experiments with SBS digitalshadow on-stage. The necessity to share a common objectbrings the scientific and technical partners to organize thesoftware platform introduced in Section 3 through shadermodules that can be easily used to build by-product soft-wares suiting artistic needs. When compared with previousad-hoc in-house approaches, it offers major improvementsin the quality of shadow extraction for artistic design, andmakes new configurations accessible for pedagogical work-shops with pupils. The GPU-based approac h to digitalshadowing of the research team made this porting possibleand opened the door to many additional special effects. Weused SBS as a representation of a semi-dependent shadowof the performers, enhanced with time delay, frame buffer-ing, and distortion effects. Blending modes between dis-tinct effects were, in this case, very useful to deconstructthe shadow and make it enter in the digital world of thevideo.

Each involved artist proposed numerous, various and spe-cific experiments to allow pupils and audience to play withtheir own shadows and discover a wide variety of creativeworlds. This creativity was also shared by librarians whobrought their own proposals to feed workshops with refer-ences and activities about shadows. We also shared ournew skills with other dance and theater companies. Takingapart the financial aspects, the cost of working with tech-nologies introduces a new parameter of power in the artsworld depending on how knowledgeable the people are withthe technology. A way for helping to overcome these powerissues is to make clearer the technological dependencies be-tween partners. It makes networking possible and resultsin stronger collaborations. We tried to find the most af-fordable hardware and software solutions for the platformin connection with a wide art-activist community includ-ing free software developers. And two companies used To-

lalusulo to work with digital shadows: Anne Morel and SansTitre Production deal with gender themes in two recentshows. Christophe Greilsammer and L’Astrolabe Cie cre-ated a performance from the famous Andersen’s tale The

Shadow. This collaboration process produces a state of mindin the use of shadows that arguably results in a new type ofexchange with the audience and with “external” artists.

5. CONCLUSION AND PERSPECTIVESThe success of the workshops leads the artists to think

about simplified derived setups on common hardware andsoftware to broadly disseminate the creative and pedagogi-cal tools. A designer will make improvements in this direc-tion. Moreover, numerous interactions already developed inthe platform by engineers and scientists offer exciting per-spectives to create new stories and performances.

Playing with the shadows implied a close relationship be-tween the performer and a screen. Attempts have been madeto break the flatness of the screen by using a 3D world inwhich shadows were walking around. This can surely bedeveloped with 3D stereoscopic projections. But an alter-native has been suggested by the scientific team to deeper

explore the possibilities offered by CSBS in interactive en-vironments using traditional lighting. This exploration willbe a future development of the current project, involving inaddition both a space designer and a light designer.

By demystifying the digital aspect of the work and focus-ing on the meaning in the art piece (even if working withtechnologies) our new practice entails necessary changes inour beliefs to be successful.

6. ACKNOWLEDGEMENTThis project has been funded by Region Ile-de-France as

a PICRI Project. We are grateful to Mediatheque Taverny,Le Cube Issy-les-Moulineaux, La Filature Mulhouse, Plate-forme Virage, and CNRS for their support. Most of allwe are very grateful to all the participants in this project:Valerie Bajcsa, Wai Kit Chan, Loıs Drougrazet, CatherineLazaro, Olivier Pfeiffer, and Renaud Rubiano.

7. REFERENCES[1] O. Bimber, A. Emmerling, and T. Klemmer.

Embedded entertainment with smart projectors. InACM SIGGRAPH 2005 Courses, SIGGRAPH ’05,New York, NY, USA, 2005. ACM.

[2] O. Bimber and R. Raskar. Spatial Augmented Reality:

Merging Real and Virtual Worlds. A. K. Peters, Ltd,Montreal, 2005.

[3] R. Casati. La decouverte de l’ombre. Albin Michel,Paris, 2002.

[4] W. K. Chan and G. Gagnere. Tllslo (toute la lumieresur l’ombre) - les configurations du dispositif.Technical report, LIMSI-CNRS, July 2010.

[5] C. Jacquemin, B. Planes, and R. Ajaj. Shadow castingfor soft and engaging immersion in augmentedvirtuality artworks. In Proceedings of the 15th

international conference on Multimedia,MULTIMEDIA ’07, pages 477–480, New York, NY,USA, 2007. ACM.

[6] M. W. Krueger. Responsive environments. In AFIPS

National Computer Conference, volume 46 of AFIPS

Conference Proceedings, pages 423–433. AFIPS Press,1977.

[7] A. Leone and C. Distante. Shadow detection formoving objects based on texture analysis. PatternRecogn., 40:1222–1233, April 2007.

[8] G. Levin. Hands up! A visual essay on the ‘Media ArtPose’, September 2006.http://www.flong.com/texts/essays/essay pose/.

[9] Y. Minomo, Y. Kakehi, M. Iida, and T. Naemura.Transforming your shadow into colorful visual media:multiprojection of complementary colors. Comput.

Entertain., 4, July 2006.

[10] R. M. Sheppard, M. Kamali, R. Rivas, M. Tamai,Z. Yang, W. Wu, and K. Nahrstedt. Advancinginteractive collaborative mediums throughtele-immersive dance (ted): a symbiotic creativity anddesign environment for art and computer science. InProceeding of the 16th ACM international conference

on Multimedia, MM ’08, pages 579–588, New York,NY, USA, 2008. ACM.

[11] Q.-Z. Wu and B.-S. Jeng. Background subtractionbased on logarithmic intensities. Pattern Recogn. Lett.,23:1529–1536, November 2002.