Introduction to Motion Tracking to Dance

Motion Tracking

Robert Wechsler

Palindrome Inter.media Performance GroupStuttgart, Germany

Lausanne, June 2009

www.palindrome.de

workshop concept

"dance first, think afterwards."

- Estragon (S.B.)

"to work with motion tracking as an artist, start and finish with the body, not with the technology."

- Robert (Palindrome)

workshop plan

1 Training. non-dancers: 11:30-11:55 ok?2 Talk / Discussion / video clips 40 min3 Improv. Chor. programming (patch-writing). 4 Tech. Sessions5 Project work6 Stretching, etc.

Talks from me (12-12:40)

Talk 1: "What is Palindrome?

A very brief history of interactive art. Motion tracking and motion capture."

Talk 2: "Sensors: body (physiological) vs. environment-based systems. mapping."

Talk 3: "Practical issues: how to get the equipment, etc. A critical discussion of video projections."

Palindrome Inter.media Performance Group

Palindrome is a kind of dance company....

• new york 82-90, germany 91-09, france?? 09-...

• performances, some installations, touring, workshops

• works with interactive music, video and lights

• develops software hardware applications*

• specialized in input and mapping

* - I'm not an engineer, but work _with_ engineers, by itself a special skill...

what is motion tracking?

• technologies that collect data on human movement (input)

• used to control sounds, music, recorded or projected text, video art, stage lighting (output)

• via performer actions / gestures / movements / bio-data

• used in interactive performances and installations

• other uses....

what is motion tracking?

• other uses:

• animation modeling (motion capture)

• scientific research (musicology, semantics, ergonomics, medicine, sports medicine, architecture)

• therapy for physically and mentally handicapped

•

motion tracking vs. motion capture

motion capture

– tracks location of fixed positions on body

– highly accurate, expensive ($200k-2m)

– generally not realtime

– used for data collection (research) and for making human or animal motion in animations (films, games, etc.) more realistic

motion tracking

– less equipment, less data, less cost (ca. $1k-2k)

– concerned with motion qualities like dynamic, direction of motion

– real time

– used for live applications: installation art, dance, theater and music

motion capture

– tracks location of fixed positions on body with reflective markers– 12-24 cameras, each lens is ringed with infrared lights

• Vicon is a leading company in motion capture

motion capture

motion capture

motion capture

• typical uses

– human movement research (sports, musicology, ergonomics, medicine,...)

– Film and Animation -- often used with 3-D animation (modeling) programs to make animations

• maya (http://www.animationarena.com)

• houdini (http://www.sidefx.com)

motion capture

motion capture

– performances of interactive animated characters: "No DNA" (http://www.nodna.com)

– experiments with dance performances

– Motione (2003-5) Arizona State U. (http://ame.asu.edu/motione)

"how long does an object linger on the end of the volume..." (Trisha Brown, Kaiser, et.al.)

"22" (Bill T. Jones, Kaiser , et.al.)– Luc Vanier et al. at the interactive performance facility U of Wisconsin at

Milwaukee (http://www.isl.uiuc.edu/Publications/final%20dance1.pdf)

experiments in real-time mo-capture

input

physical

human action

motion tracking and interactivity

media output

sounds,

music, text,

projections,

lighting

input

physical

human action

motion tracking

computersensor

(e.g. video camera)

output device (e.g. loud speakers)

media output

sounds,

music, text,

projections,

lighting

input

physical

human action

motion tracking

computersensor

(e.g. video camera)

output device (e.g. loud speakers)

media output

sounds,

music, text,

projections,

lighting

analogue to digital

conversion

digital to analogue

conversion

digital vs. analogue analogue data

• hard to reproduce• "rich data" (infinite values)• very high resolution• more details• contaminated data (becomes noisy, but rarely fails completely)

digital data

• easy to reproduce• lower resolution, less „human-feel“• easy to store• easy to process• contaminated data remains clean (errors can be filtered) or signal fails altogether

input

physical

human action

motion tracking

sensor (e.g. video camera)

output device (e.g. loud speakers)

media output

sounds,

music, text,

projections,

lighting

computer

What does motion tracking have to do with interaction or art?

interaction means

human being 1 human being 2

• a back and forth of energy, impulse and ideas

Drawings. France and Spain ca. 3000 BC

interactive performance is nothing new

-- you don't need computers to do it!

interaction

who with whom? what with what?

• between performers ?

• between performer and audience ?

• between audience members ?

• between humans and machines ?

interaction vs. automation

processing

interaction

• is an integral part of human life

• is instinctive, deeply felt, highly

communicative

• can and usually does occur on

many sensory levels

• is the heart and soul of live

performance

automation

• if new, excites curiosity

• feels like "future vision"

• hugely popular in film, tv... processingaction re-action

human action human re-action

a very brief history of technology-supported interaction in performance

Dionysis Thr. Athens - 3c BC

Ancient greek theater employed a number of mechanical devices to extend the

power of the actor to allow them to play gods and goddesses.

Louis XIV, the Sun King, was fascinated by remote-acting mechanical devices

on the battle field. Cannons allowed you to take deadly action from a distance

(500 meters). "gesture amplification". (not an interactive device...)

Electronics, interactive sound and lighting devices were used by New York performance artists in the 60's and 70's

Robert Rauschenberg - "Pelican" 1963

Laurie Anderson - "Walking Solo for Tape Bow Violin" 1979

reasons to use motion tracking

• amplification of gesture

• makes possible new ways of associating and connecting media

• _can_ be interactive

• sense of spontaneity and engagement

• environment becomes lively and responsive

• text and subtext (body can speak, as well as mouth)

• synesthesia

• the technology itself is catchy. marketable.

end of part 1.

www.palindrome.de

Motion Tracking - part 2

Talk 2: sensors mapping eyecon instruction Talk 3: what you need to track motion different software and hardware available where to buy it cheaply video projection techniques artistic issues

robert wechsler www.palindrome.de

motion tracking sensors

• body-oriented

• environment-oriented

motion tracking sensors

• body-oriented

– body-worn electrodes

• EEG

• ECG

• EMG

• Touching

– joint bend-sensors

– accelerometers, other body-worn devices

• environment-oriented

– video camera-based (eg. eyecon)

– infrared camera-based

– floor sensors, wall sensors, etc

mapping

cause and effect. what does what.

mapping has 3 parts:

• input

• output

• logical relationship

input parameter relationship/processing output parameter

components of mapping

relationship/processing

• logical direction• compliance• transparency (clearness)• diffusion (subtlety)

mapping

mapping choices multiply:

input parameters x output parameters x logical relationships

= number of mappings

there are important human perception issues in these choices!

some can be perceived, many can not!

video

1. body-part locations (presence /absence), e.g. body parts

2. movement dynamic (in defined zones)

3. movement direction (left-right, up-down)

4. size factors (height, width)

5. position in room (or stage)

bio-sensors• physiological properties (muscle contractions, joint bendign, etc.• contact with objects or other people

mapping

"useful" inputs

internal • audio synthesizer (on/off, volume control, pitch bend, panning)• audio samples (on/off, volume control, pitch bend, panning)• images, videos (on/off, scrubbing)

external (secondary machines)• sound programs (soundforge, MAX/msp, ...), algorithmic composition• controlling stage lighting (DMX) • realtime graphics (Director, Flash, Isadora, Kalypso, Nato*, Jitter*, SoftVNS*,

SquishedEyeball*, ...) • mechanical devices

* objects that interact with cycling74‘s „Jitter“, the graphics processing environment for MAX/msp.

mapping

"useful" outputs

Boolean (logic in on/off relationships)

•  positive: movt.=media, no movt.=no media

•      negative: movt.=no media, no movt.=media

•  toggle: movt.=media, 2nd. movt.=no media

• iteration: movt.=media, 2nd movt.=2nd media, etc.

•   random, sequential or simultaneous effects

continuous controllers

•  higher/lower – as in a slider (loudness, pitch, filters, etc.)

•    thresholds – beyond which a certain change occurs

•  

mapping

"useful" relationships

motion tracking systems

• EyeCon (http://www.eyecon.palindrome.de)

• Isadora (http://www.troika.org)

• Eyes Web (http://www.eyesweb.org)

• Very Nervous System (http://homepage.mac.com/davidrokeby/vns.html)

• Cyclops (http://www.ericsinger.com/cyclopsmax.html) *

• CV.jit (http://www.iamas.ac.jp/~jovan02/cv/) *

• MIDI Dancer (http://www.troika.org)

• Big Eye (http://www.steim.org)

* - programmed in Jitter the graphics processing environment for MAX/msp.

end of part 2.

robert wechsler www.palindrome.de

Motion Tracking - part 3

Talk 3: what you need to track motion different software and hardware available where to buy it cheaply video projection techniques artistic issues next steps...

Eyecon Training how to control images and video how to connection eyecon to MAX/mp

Project Work

robert wechsler www.palindrome.de

what you need to track motion Of course there are different ways to do it. Different artists have different styles

of working, different needs...

Advantages of EyeCon

flexibility, sensitivity -- you can use it in many different ways

ease of use, quick start-up time, DIY (do it yourself)

relatively low cost

To use EyeCon in the "best way", requires 4 things:

computer running windows

software

camera

industrial framegrabber*

what you need to track motion Computer needs video input

Digital video (Firewire, USB2)

+ digital cameras (camcorder, webcams)

+ low noise

+ works with laptops

- latency issues

- image resolution issues (smaller chip sizes)

- limited cable length

Analog video

+ "unlimited" cable length

+ lower latency

+ even digital cams usually have analog output

- cost more (although many older cameras work quite well)

- works less well with laptops... (?) i.e. need an external or internal framegrabber

what you need to track motion Framegrabber

captures, or digitizes video (turns analog video into digital video)

desktop allows for industrial framegrabber:

must be: Falcon (LeoLite, et al.) www.ids-imaging.com

what you need to track motion Camera

almost any camera will work, but...

chip size 1/2" allows WIDE angles and high resolution

does not need to be color

what you need to track motion Lowest Costs

desktop computer

get a used one? min. 1 GHz 300 e

new, shuttle, barebones, 4 GHz 600 e

framegrabber 250 e

camera 1/2" chip 350 e

zoom lens (4-12 mm) 75 e

software licenses 100 – 500 e

Total Equipment Cost: 1000 – 2000 e

Training (w/ me :) ) includes building equipment

1 week 1000, 3 day intensive 600

motion tracking systems

• EyeCon (http://www.eyecon.palindrome.de)

• Isadora (http://www.troika.org)

• Eyes Web (http://www.eyesweb.org)

• Very Nervous System (http://homepage.mac.com/davidrokeby/vns.html)

• Cyclops (http://www.ericsinger.com/cyclopsmax.html) *

• CV.jit (http://www.iamas.ac.jp/~jovan02/cv/) *

• MIDI Dancer (http://www.troika.org)

• Big Eye (http://www.steim.org)

* - programmed in Jitter the graphics processing environment for MAX/msp.

motion tracking - issues for artists• Artists use technology differently than scientists and engineers! They have different needs, different criteria.

1 working with engineers

- higher tech, flashier effects

- better control, more reliable

- less problems? probably not. different problems...

- you will find yourself making work _about_ technology – whether you intended to or not!

2 working alone

- easier to focus on artistic issues

- simpler systems

- less flashy (= less attention, less money)

- requires tons of patience

movies, sounds and lights

Media can compliment and support the live performer

or it can distract from her-him.

This means it depends on how you do it.

And there is a fundamental issue to think about...

1. make the image smaller

2. make the image less bright, or black & white

3. lower the resolution of the film, "pixilate" it or use various image abstraction methods

4. use non-human material -- human images and faces distract most of all

5. use semi-transparent screens

6. put the screen up high, down low, in the back, or use thin materials

7. project on the side walls, on the floor or ceiling

8. project on the bodies of the performers

9. have the projection incorporate the performer's live image

10. put the image under the control of the of the performer's movement (ie. make it interactive)

movies (moving images) are extremely distracting to the live performer.

APPROACH 2:

Use Stills. Low Motion Slow Motion.

• Palindrome Intermedia Performance Group• Krisztina de Chatel• Igloo• Ventura Dance (Pablo Ventura)• Robert Lepage • André Werner • Marlon Barrios Solano• La la la Human Steps• Georg Hobmeier• Leine Roebana Dans Kompanie• Troika Ranch• Blue Man Group• you

who is using motion tracking?

motion tracking - review

Positive

amplification of gesture

sense of spontaneity and engagement

text and subtext (words say one thing, body says another)

can make art interactive in new ways

generates genuinely new ways of associating and connecting media

helps to get grants and invitations

Negative

substitute art with special effects

complex processes -- compromise quality and subtlety of expressive media.

extremely time consuming, distracting

life-less materials and tools de-humanize art and artists

Motion Tracking

End of part 3.

[email protected] www.palindrome.de

"dance first, think afterwards."

- Estragon (S.B.)

eyecon www.eyecon.palindrome.de