Carleton Motion Capture (CMC) Matt Kracum, Will Levine, Coryn Pavelsky, Kohei Shinkawa Jack Goldfeather.

Carleton Motion Capture(CMC)

Matt Kracum, Will Levine, Coryn Pavelsky, Kohei Shinkawa

Jack Goldfeather

Project Goal

• Record a person’s movements– Begin with simple gestures– Store gestures in a computer-readable way

• Display them as naturally as possible in 3D– User interface– Begin with 2D

Tasks for 2D

• Capturing data– Video camera– Mark body parts– Reduce other information

• Media formats– Video file split into frames

Tasks for 2D

• Tracking algorithm– Computer works frame by frame to identify and align points

• Animating and presenting data– Java Open Graphic Library (JOGL)– User interface

Tracking Methods

Stickies Color LEDs White LEDs

2D Marker Tracking

• Builds list of coordinates of each marker in each frame

• Built for colored sticky notes, repurposed for white LEDs

2D Scanning Algorithm

For each picture:For each expected marker:

For each pixel, starting where previous marker left off:If pixel above a threshold:

Mark it, update some data, recursively search neighbors

Save marker’s coordinates

Finding Marker Coordinates



Min X, Max XMin Y, Max Y


Min X

Max X

Max Y

Min Y = (Average X, Average Y)

Determining center:

Save these values, run again from stopping point to find another marker

Processing Coordinates & Adding Predictions

• Huge mess of data• Must reorder coordinates• Predict missing points based on previous

frames

Reordering Coordinates

Picture 1 Head L. Shoulder R. Shoulder Foot


Picture 3 Head R. Shoulder L. Shoulder Foot

Double array of coordinates:




Picture 3 Head R. Shoulder L. Shoulder Foot







Predicting Coordinates



Picture 3 Head L. Shoulder Foot ????





Picture 3 Head L. Shoulder ???? Foot





Picture 3 Head L. Shoulder R. S. Guess Foot


R. S. Guess = R. Shoulder2 + (R. Shoulder2 – R. Shoulder1)

Send or Save Data

• Send double array to Animation Panel• Save data to a file

Out File

Double Array

Marker Tracking Animation Panel

OR

2D Visualization

• Run video & animation in parallel at 30 fps• Find proper Image format• Creating animation from data

Objectives

Display image at 30 FPS

• 30 FPS = 1/30 second per frame– Roughly the minimum rate human eyes can

not observe any flickering

• FPSAnimator – OpenGL Subclass can achieve a target

frames-per-second rate to avoid using all CPU time.

In 1/30 sec the program has to do…

• Animation– Figure where each point is from data file– Draw points and lines to show the corresponding stick man

• Video– Load a image file from directory– Render image to the panel

If even one of the processes costs more time, everything will be delayed

How do we run 2 threads in sync?

• SemaphoreAllow thread A to update its image only when semaphore is down. Likewise, thread B can only update its image when it’s down. Thread B after successful update increases the current position by 1.

Find proper Image format

• Two candidates for the image format– bitmap• Lossless compression• High quality

– jpg• Lossy compression• Low quality

For our image displaying, the quality of each image didn’t really matter.

Creating animation from data

• All the vertex information was stored in comma-separated value (CSV) file format

• Information is pre-ordered during the tracking phase

• Load all the information prior to drawing animation to reduce the I/O cost

Working with GUI

• Deploying Model-View-Control (MVC) model

•View: Two threads drawing images to panels. Check the model class every time it update the display.

•Model: Wrapper class shared among threads and controller class

•Controller: Directly listens to the GUI. Updates the model upon calls from the GUI

2D Demo

Challenges

• Recording & Processing– Marker changes– Thresholding– Camera limitations– Bounding, disabling guesses

• Animating & GUI

Tasks for 3D Upgrade

• Upgrade marker tracking– Use the ‘Z’ coordinates

• Implement math– Reconciling multiple camera views– Linear algebra

• Implement 3D animation

Tasks for 3D Upgrade

• Upgrade GUI– Change perspective in 3D model– Keyboard and mouse listeners, changing matrices

• Film for 3D– Synchronize cameras– Determine camera positions

Room Setup

Room Setup

Room Setup

Room Setup

Room Setup

Room Setup

3D Math

• Find world coordinates using 2 cameras• Find camera rotation matrix• Find camera translation vector

General

General

General

Problem:

Each camera has a different coordinate system from the

world.

General

General

Solution:

Translate and rotate the coordinate systems to match.

Rotation

Rotation

Rotation

Rotation

We know that some rotation matrix R times V is in the plane of v. Thus we

get the equation:

n . (R*V) = 0unknown

Rotation

V is a vector parallel to the x axis, thus it can be represented as (s, 0, 0).

r11 r12 r13r21 r22 r33r31 r32 r33

( ( s00) =( r11

r21r31)R*V =

Rotation

n . = 0( r11r21r31)

a*r11 + b*r21 + c*r31 = 0

We have 3 equations and 3 unknowns, solve for the unknowns. Repeat for

each axis.

Translation

Translation

Translation

Since (T – P) is in the same plane as v and we know the normal to that

plane, we get the equation:

n . (R*(T - P)) = 0UnknownT = (t1, t2, t3)

Translation

This simplifies to:

a*t1 + b*t2 + c*t3 = 0X translate Y translate Z translate

We have 3 equations and 3 unknowns, solve for the unknowns.

General

General

Now they are in the same coordinate system so we can find

an intersection.

Rotation

Calculate the rotation from world coordinates to camera coordinates using known vectors parallel to the

world coordinate system.

Translation

Calculate the translation from world points to camera points

using our calculated R and known points.

3D Coordinates

Find the 3D coordinates of a point seen through 2 or more cameras

using our translation and rotation.

3D Coordinates

3D Coordinates

The world point must be somewhere along W, the equation

for it is:

WC = T - s(R-1*c)Unknown

Scalar for WUnknown

World coordinate

Translationvector

Inverse rotationmatrix

Camera vector

3D Coordinates

3D Coordinates

We construct the equation:

(WC1 - WC2)2 = 0

(T1 - s1(R1-1*c1)) - (T2 - s2(R2-1*c2))2 = 0

OR

3D Coordinates

Take the derivative with respect to the 2 unknowns s1 and s2 which is:

a1*s1 + b1*s2 + c1 = 0

a2*s1 + b2*s2 + c2 = 0

Errors

• No solutions to equations for rotation and translation due to human error.

• Using least square sum we minimize instead of solving.

• More math, conceptually the same.

3D Marker Tracking

• Upgrade to 3D:– Input from three cameras– Output real world coordinates

• Fix problems with 2D:– Messy– Inaccurate– Inflexible

Upgrade to Mkr Objects

• Linked-list of these• Stores coordinates & other info• Allows easy prediction• Keeps count of guesses• Disable flag• Easily finds head and last known markers

Linked-List Based Algorithm:

Save marker’s coordinates into a Mkr object, find and link to previous marker’s Mkr object, and fill in missing markers

Linked-Lists of Coordinates

Head, picture 1:X, Y, Z coordinatesHead, picture 1:X, Y, Z coordinates



Left Shoulder, picture 3:X, Y, Z coordinatesLeft Shoulder, picture 3:X, Y, Z coordinates



Picture 2Coordinates

Camera 1Camera 1

hhhhh

Reordering & Missing Markers

HeadHead

HeadHead

HandHand

HeadHead

HandHand

FootFoot

FootFoot

FootFootPicture 3coordinates

Camera 1Camera 1

hhhhh


HeadHead

HeadHead

HandHand

HeadHead

HandHand

FootFoot

FootFoot


Just change pointers!

Camera 1Camera 1

hhhhh


HeadHead

HeadHead

HandHand

HandHand

FootFoot


Mkr objects keep track of levels, so easy to find what’s missing

HeadHead FootFoot

Camera 1Camera 1


HeadHead

HeadHead

HandHand

HandHand

FootFoot

FootFootPicture 3coordinates HeadHead FootFoot

HandHandHandHand

HandHand

Camera 2Camera 2 Camera 3Camera 3

World CoordinatesWorld Coordinates

HandHand

Rotate & translate back

Camera 1Camera 1

hhhhh


HeadHead

HeadHead

HandHand

HandHand

FootFoot

FootFoot

GuessGuessPicture 3coordinates

-Otherwise, predict from previous Mkr objects in list & increments guess count-If guess count too big, revert to last known coordinates

HeadHead FootFoot

Camera 1Camera 1

Creating World Coordinates

HeadHead HandHand FootFoot

HeadHead HandHand

HeadHead FootFootHandHandFrame 1


Camera 2Camera 2

HeadHeadFrame 1


FootFoot

Creating World Coordinates

HeadHead HandHand FootFoot

FootFootHandHand

HeadHead FootFootHandHandFrame 1


Camera 2Camera 2

Frame 1


HeadHead

HeadHead FootFootHandHand

Repeat for Next Frame

Frame 1


HeadHead FootFootHandHand

HeadHeadFrame 2

3D Visualization

• Constructing 3D world• Obtaining information from .obj files• Attaching models properly

Constructing 3D world

– Modelview • Modelview transformations are used to position the

items in the scene.

– Projection• Projection transformations defines how the 3D scene is

projected in the 2D screen.

Obtaining models from .obj file

• Dissect polygons into groups to form different body parts

• Assign different color coordination to each group

Attaching models properly

• Algorithm1. Draw models with size = 1 on (0,0,0) coordinates

aligned to z-axis2. Move model to sit on the lower point3. Adjust model’s coordinate to the vector formed

by the two points and its orthogonal vectors 4. Scale model to fit to the actual size

Draw models with size = 1 on (0,0,0) coordinates aligned to z-axis

Move model to sit on the lower point

Adjust model’s coordinate to the vector formed by the two points and its orthogonal vectors

Cond. Adjust model’s coordinate to the vector formed by the two points and its orthogonal vectors

Mathematics for moving objectFirst Vector (PO)

Z.x = P.x - O.xZ.y = P.y - O.yZ.z = P.z - O.z

Second Vector (orthogonal to PO)Y.x = Z.yY.y = -Z.zY.z = 0

Third Vector (Cross product of first and second vector)X.x = Y.y * Z.z - Y.z * Z.yX.y = Y.z * Z.x - Y.x * Z.zX.z = Y.x * Z.y - Y.y * Z.x

Final MatrixX.x X.y X.z 0Y.x Y.y Y.z 0Z.x Z.y Z.z 00 0 0 1

3D Demo

For the Future

• Better equipment– Cameras– LEDs

• Improved tracking algorithm– Predictions– Bounding

For the Future

• Independence from Mathematica– Porting equation code to Java

• Wish: real time streaming!– Computer would show you as you move.

In the Real World

In the Real World

Carleton Motion Capture (CMC) Matt Kracum, Will Levine, Coryn Pavelsky, Kohei Shinkawa Jack Goldfeather.

Documents

shoulderfoot picture

marker slide

d slide

predicting coordinates

reordering coordinates

marker coordinates

shoulder1 slide

max y slide