Shadow Figures: An Interactive Shadow Animation Platform ...interactive live performance using computer vision tracking and transition techniques. 2.2 Infrared Hardware An infrared

Shadow Figures: An Interactive Shadow Animation Platform for

Performance

B. Tyler ParkerBrown University Computer Science Department

Providence, RI 02912

May 2013

Abstract

While front projection is often a viable solution forthe creation of large displays, it is limiting for aninteractive context. A user standing in front of aprojected display results in a shadow and occludedimagery. This paper demonstrates an effective ap-proach for eliminating user shadows and occlusionin a front projection setup. It further elaborateson this approach by presenting new software toolsto replace and augment a performer’s shadow withpre-recorded and generated imagery for interactiveand performative purposes. Despite issues arisingfrom projector latency and calibration imprecision,an effective proof-of-concept system for interactiveshadow performance was created.

1 Introduction

1.1 Problem Statement

Typically, projectors are used to display imageryon a flat surface for a passive audience. However,problems arise if the projections need to be dis-played in an interactive context. The primary is-sue is occlusion: if a user stands in the way of theprojection, it creates not only a shadow behind theuser, but a distorted image projected on the user aswell. Multiple projectors can be used to alleviatethe shadow problem by creating multiple but lessnoticeable shadows. Performers may wear clothingthat either blends with or does not reflect the lightprojected on them, but this also only goes so far.

(a)

Figure 1: Even the acclaimed Metropolitan Opera productionof Siegfried [1] encountered occlusion issues with their projectionmappings. Spotlights are used not only to draw focus but to washout the projection cast on a performer, eliminating the imageryprojected on them but also creating a stark shadow.

A possible solution would be to have a systemthat detects an occluder and eliminates both theunwanted light projected on the occluder and itsshadow.

Detecting, tracking, and removing a shadow inreal-time introduces opportunities for innovativeinteraction and performance. The replaced usershadow (either pre-recorded or generated) can be-come an independent character. To achieve thiseffect, custom tools and techniques would need tobe developed, with the constraint that they mustperform in real-time. Computationally restoring ashadow also poses a unique problem for the shadowremoval algorithm, as it needs to recognize a real

Shadow Figures: An Interactive Shadow Animation Platform for Performance 2

shadow from a generated one and be unaffected byany other projected imagery.

1.2 Background and Previous Work

The shadow removal problem has been approachedin several different ways. One trend is to track anobject in a video and use color comparison near thatobject to determine what pixels belong to the dark-ened background of its shadow. Both [2] and [3]employ this technique and remove shadows in real-time. Unfortunately, they are too noisy in theirresults. For these methods to work a near-perfectshadow must be detected and recorded. In addition,having a dynamic background as well as a generatedshadow would greatly increase the difficulty of thistask. For the purposes of enabling the algorithm torun in real-time, and to have a clean shadow for thesake of the illusion, working with traditional visiontechniques in the visual light spectrum appears tobe an overly problematic and ultimately ineffectiveapproach.

(a) (b)

Figure 2: Both [2] and [3] have strong artifacts when detectingand segmenting a shadow from the background.

An infrared-based approach circumvents some ofthe previously discussed problems with visible light.In this case, a projected dynamic background is nolonger a problem because it would not affect an in-frared reading, and an artificially generated shadowalso would not pose a problem for the same reason.In [4], a setup that would potentially be very ef-fective in shadow elimination is demonstrated. Thepaper purely focuses on the front projection prob-lem (where the projected light “blinds” the pre-senter) without extensively discussing shadow re-moval. Useful solutions to certain sub-problems of

the project (such as multi-projector image align-ment and calibration) are covered by [5], yet involvean overly complex tracking model to eliminate shad-ows. The approach used by [6] for the most part ad-dresses the needs of the shadow removal componentof this project: it is infrared-based which mitigatesthe issues with a dynamic background and gener-ated shadow, it operates in real-time, and results ina clean shadow image.

2 Description

2.1 Approach

Based on the previous work, an infrared-based ap-proach to the shadow removal/detection problemwas used. By occluding an infrared lit backdrop,a user’s silhouette is recorded via an infrared cam-era. This image, effectively a user’s shadow, is thenused to generate occlusion masks to create two seg-mented feeds. Using two projectors whose projec-tions are mapped to the same surface, the two seg-mented feeds are overlaid to create one unified pro-jection that eliminates the shadow.

For the interactive and playback portion, prere-corded shadows and imagery are integrated with theinteractive live performance using computer visiontracking and transition techniques.

2.2 Infrared Hardware

An infrared floodlight is created by taking a normallight and using two colored light filters: Congo Blueand Primary Red. Congo Blue blocks green, yellow,and most red light, while Primary Red blocks bluelight.

Overlaying the filters effectively multiplies theirwaveforms, blocking the visible spectrum but allow-ing infrared light to pass through. For this setup,3 Congo Blue filters and 1 Primary Red filter for asingle light source were used [8]. Additional CongoBlue gels were required as they reduce, but do notcompletely eliminate, red light (see Figure 3). Onefloodlight with a modest light throw was assembledfor the purposes of the proof of concept, more in-

Shadow Figures: An Interactive Shadow Animation Platform for Performance 3

(a) (b)

Figure 3: Light transmission graphs for the two light gels [9].Cobalt Blue (a) blocks out green and most red, Primary Red (b)blocks out blue and green. Combined, they block out most of thevisible light spectrum, allowing infrared light to pass through.

frared lights covering the scene can scale this setupto an arbitrarily large performance space.

Creating an infrared camera utilizes the sameprinciples of creating an infrared light. All digitalcameras contain a filter that blocks infrared light,otherwise this light would interfere with the imagesensor. By replacing the infrared filter of a web-cam with the same combination of light gels onlyinfrared light reaches the image sensor, resulting ina single-channel infrared digital image.

(a) (b)

(c) (d)

Figure 4: The disassembled camera with its image sensor (a) andthe filters used to block out visible light (b). The same sceneviewed with a normal visible light camera (c) and an infraredcamera (d).

2.3 Occluder Mask Generation

Each frame recorded from the IR camera is pro-cessed to generate two images: an occluder and anon-occluder mask. These two masks are used tosplit the desired background image into two com-plementary projections. Overlaying these two pro-jections on one background surface results in a sin-gle, unified background projection, despite occlud-ing agents (such as a performer).

The first IR frame recorded is saved for differenc-ing subsequent frames in order to obtain the per-former’s silhouette. At this point, the software canbe put in recording mode and save out the differ-enced frames for later use as a shadow performance.

The current “shadow frame” is thresholded andthen dilated, resulting in a binary image with abuffer around the occluder. This buffer providesan important margin for error when tracking andreplacing a moving shadow, mitigating the effect ofthe various latencies in the system (computationaltime per frame, IR camera FPS, projector displaylag, etc.). The now-dilated image is then blurredin order to blend visible seams when overlaying pro-jector segments. The resulting image and its inverseserve as the occluder and non-occluder masks.

2.4 Projector Roles

Once these masks are generated, they are imagemultiplied with the desired background image in or-der to create two piecewise projections.

The first, or source, projector is treated as the pri-mary light source. It projects the segment of the de-sired background that is generated by image multi-plying the background image with the non-occludedmask. The IR camera has been placed as near tothe source projector as possible; this allows for aperformers detected IR silhouette to effectively actas a detected shadow.

The second, or fill, projector fills in the missingsegment left behind by the source projector. Its pro-jection is created by multiplying the background im-age with the occluded mask (the non-occluded maskinverse).

Warping the homographies of the two projectors

Shadow Figures: An Interactive Shadow Animation Platform for Performance 4

(a) (b)

(c) (d)

(e) (f)

Figure 5: IR camera frame (a), differenced shadow (b), thresh-olded (c), dilated (d), blurred/non-occluding mask (e), in-verted/occluding mask (f).

(a) (b)

Figure 6: The resulting projected piecewise background image.

in order to map their output to the same surface re-sults in a unified piecewise projection. In this man-ner, the shadow caused by the occluder is effectivelyeliminated.

At this point the project closely matches the Vir-tual Rear Projection setup described in [6]; thissetup can be used to create a large display, wherehaving an actual screen or rear projection sourcewould be impractical due either to space or costconstraints. Yet this project does not use this tech-

(a) (b)

(c)

Figure 7: Projections of the calibration images (a) and (b) arealigned by using the software to distort their homographies, cre-ating a seamless overlapped projection.

nology as an end-all, but rather as a springboard forartistic performance and interactive applications.

2.5 Shadow Replacement/Tracking

With the performer’s shadow removed, the pro-jected background is a blank slate which can hostany arbitrary imagery. The removed shadow caneither be computationally added back in, swappedwith a previously recorded shadow, or replaced withany other generated content.

By tracking both the removed and recordedshadow’s positions, the recorded shadow can haveits translatory motion match the current perfor-mance. This allows a recording to follow a live per-former, creating an interactive dynamic.

Additionally, shadow sequences can be swappedon the fly. In order to blend the transition betweenlive and recorded frames in real-time, the contoursof both shadows are computed and the vertices ofthe live shadow contour are mapped to the verticesof the recorded contour. The resulting “tween” con-tour morphs between the shapes over a given lengthof transition frames. During this transition, thetween contour is filled and slightly blurred to giveit the appearance of an intermediary shadow. Thiscreates a smoother transition than a jarring jump

Shadow Figures: An Interactive Shadow Animation Platform for Performance 5

from one potentially disparate shadow to another.

(a) (b)

(c)

Figure 8: Tweening from the performer’s shadow to the replace-ment imagery (a bat).

Tweening and tracking are just two applicationsfor the shadow detection functionality. Shape de-tection could be used as triggers for animated se-quences, or as drivers for interactive characters suchas virtual puppets [11].

2.6 Performer/Shadow Lighting

One technique used in other shadow performance in-stallations is to have a shadow contain imagery andanimation, such as in Matreyek’s Glorious Visionsperformance [7]. While Matreyek relied on chore-ographed motion and poses to achieve the illusion,one can use shadow tracking to achieve this effectwith any spontaneous live performance. Using anon-thresholded frame of a computed live shadowor a previously recorded shadow as a mask, as wellas optionally using position tracking information, ashadow performance can appear to contain any ar-bitrary imagery.

Similarly, any imagery or animation can betracked to and projected on the performer. Nor-mally an occluder would have no light from eitherprojector cast on it, as this would cast a shadow (de-feating the purpose of the entire setup). By erod-ing and blurring the thresholded image of the per-

former’s shadow, a mask that is slightly smaller thanthe performer’s silhouette is obtained. That, com-bined with the desired imagery, is projected directlyonto the performer. Yet, unlike the occlusion masks,this mask has very little margin for error and re-quires a near fit to be effective; the mask has to beclosely matched to the performer at all times. Asignificantly smaller mask would not cover the per-former effectively yet a larger mask may overlap thesilhouette and spill light onto the background, ruin-ing the illusion.

3 Results

A user interface for adjusting shadow re-moval/replacement properties was implemented,as well as the ability to save/load a settings file.Balancing the settings was key for an effectivemask.

(a)

Figure 9: The user control panel, able to tweak such settings asdilation, blurring, and playback states.

Dilation and blurring of the masks proved to bethe user-controlled properties that had the largesteffect on the framerate of the application; too manyblur and dilation iterations would lower the numberof frames per second considerably. Increasing themask would provide a larger margin for error, buttoo large a mask would increase latency, causing ad-ditional error, as well as overlap the occluding andnon-occluding projections.

Blurring the masks was crucial for the blending

Shadow Figures: An Interactive Shadow Animation Platform for Performance 6

of the mask seams in this setup. Minor imperfec-tions in projector calibration caused a slight dark-ened border to appear around the mask segments,but this was acceptable for the scope of this project.Further methods for the mitigation or elimination ofthis artifact are discussed below.

With a well balanced set of properties the soft-ware algorithm could run upwards of 60 frames persecond (FPS), typically between 40-50 FPS with ad-ditional tracking and shadow replacement function-ality activated. With full performer and shadow an-imations it ran around 30 FPS. All these frameratesare within acceptable bounds for the illusion of con-tinuous motion.

While the software was relatively fast, the projec-tors and camera used for the project had a distinctlatency, and as such, if the performer was movingrapidly, the performer could exceed the projectors’capacity to compensate and remove the shadow.This effect was even more pronounced with frontprojection on the performer, where the margin forerror is narrower. In addition, a stuttering effect wasvisible with fast motion. For this specific hardwaresetup, these limitations must be kept in mind; anemphasis on restrained user motion but highly dy-namic shadow animation could be an effective wayto work with the issue.

For a full demonstration of the shadow tool’s func-tionality, please view the accompanying video.

4 Discussion and Future Work

Building on previous strategies for shadow removalusing infrared background illumination and occlu-sion masking, a new system for interactive perfor-mance was developed and a proof-of-concept suc-cessfully implemented, providing: 1. a means tocalibrate two projector planar homographies, 2.processing of an infrared camera feed to isolatea performer silhouette, 3. recording of a com-puted shadow, 4. contour/position tracking of ashadow, 5. elimination/replacement of a shadow,6. shadow transition tweening, and 7. animationoverlay/tracking/substitution for either the live per-former or recorded/generated shadow. All this func-

tionality serves as a foundation for interactive instal-lations and performance, and could be extended ina myriad of ways.

One way to extend this project would be to re-fine projector mapping. The project scope assumedthat user distorted projector-surface homographieswould be an effective means of calibration, but acomputational solution would be even more accu-rate. Techniques also exist to accommodate for thesubtle lens radial distortions of a projector, as wellas for differences in brightness (such as using Lumi-nance Attenuation Maps (LAMs) [6]). Planar dis-tortions also do not take into account a non-uniformplanar surface; an interesting extension would be topre-calibrate the projector output to warp to anyarbitrary 3D surface geometry.

The primary bottleneck in the setup was not thealgorithm but the optical hardware. With the appli-cation running upwards of 60 frames per second, theIR camera could only reach half that (with a lowerresolution setting). The projectors appeared to haveeven less effective FPS, with chromatic aberrationbecoming a problem when compensating for anysufficiently speedy motion of the performer. Muchof this could simply be fixed with better hardware.There are faster cameras and projectors specificallydesigned for outputting the required frames per sec-ond necessary to virtualize reality. However, thislevel of technology would have been outside the bud-get of a student’s DIY proof-of-concept, which wassufficiently demonstrated even with the hardwareconstraints.

A possible extension that could mitigate the“frame-stuttering” of fast motion would be to dy-namically generate a motion blur that visuallymatches the projector frame-rate (if the camera con-tribution of motion blur is not enough). Addition-ally, motion prediction could be employed to gen-erate the occlusion masks and computed shadows.Synchronizing the projectors might also alleviate thestuttering.

As for future applications of the interactive andartistic potential of the technology, they are prac-tically unlimited. Detected shadows could be usedas a means to play and engage [11] or as merelyan interface for interaction with large displays [12].

Shadow Figures: An Interactive Shadow Animation Platform for Performance 7

Performers may use the technology for visual effectand exploration of utilizing a shadow as a comple-mentary character.

Ultimately, the aim of this project was to imple-ment and showcase a technological setup and toolsetthat could be used for a large variety of applications,be it for artistic expression or practical interactivesolutions.

Acknowledgments

I would like to especially thank Barbara Meier forher mentorship and support. I also thank ToddWinkler for his helpful reference and discussion re-garding the hardware component of the project.

References

[1] Lepage, Robert. Der Ring des Ni-belungen - Robert Lepage. Retrieved4/25/2013, from http://realisations.

net/Robert-Lepage-MET-Opera-NYC-2011

[2] D. Grest, J.-M. Frahm, and R. Koch, A colorsimilarity measure for robust shadow removalin real time, in Proc. of Vision, Modeling andVisualization (VMV), Munich, Germany, Nov.2003, pp. 253260.

[3] Mohamed Dahmane,Jean Meunier. Real-timemoving object detection and shadow remov-ing in video surveillance[J]. 3rd InternationalConference: IEEE of Sciences of Electronic,Technologies of Information and Telecommuni-cations March 27-31,2005.

[4] D.S. Tan, and R. Pausch. Pre-emptive Shad-ows: Eliminating the Blinding Light from Pro-jectors. In Interactive poster at CHI 2002 Con-ference on Human Factors in Computing Sys-tems, Minneapolis, MN, 2002.

[5] S. Audet and J. R. Cooperstock. Shadow Re-moval in Front Projection Environments UsingObject Tracking. In Proceedings of the 2007IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition (CVPR

07) - Workshops (Procams 2007). IEEE Com-puter Society, 2007.

[6] Matthew Flagg, Jay Summet, and James M.Rehg, Improving the speed of virtual rear pro-jection: A GPU-centric architecture, in Pro-ceedings of the 2005 IEEE Computer SocietyConference on Computer Vision and PatternRecognition (CVPR05) - Workshops, 2005, vol.3, p. 105.

[7] Matreyek, M. (2010, July). Miwa MatreyeksGlorious Visions [Video file]. Retrievedfrom http://www.ted.com/talks/miwa_

matreyek_s_glorious_visions.html

[8] Beaty, Bill. INFRARED GOGGLES FORUNDER $10: A Human IR Vision Exper-iment. Retrieved 5/14/2013 http://amasci.

com/amateur/irgoggl.html

[9] Lee Filters’ color details for PrimaryRed #106 and Congo Blue #181, fromhttp://www.leefilters.com/lighting/

colour-details.html

[10] Sibbe, Scott S. (n.d.). Scott Sibbe: InteractiveArt. Retrieved 5/13/2012, from http://www.

snibbe.com/

[11] Design I/O LLC, PUPPET PARADE. Re-trieved 4/20/2013, from http://design-io.

com/projects/PuppetParadeCinekid/

[12] Shoemaker, G., Tang, A., and Booth, K.S.(2007). Shadow reaching: a new perspective oninteraction for large displays. Proc. UIST 2007,5356.