The Implementation of Markerless Image-based 3D Features Tracking System Lu Zhang Feb. 15, 2005.

The Implementation of Markerless Image-based 3D Features Tracking System

Lu Zhang

Feb. 15, 2005

Motivations

Objective Find more efficient algorithms to implement 3D volume tracking

based on 2D Image sequences.

Problems in this topic 1. Huge datasets For only one data file: 128*128*128 2. Computation time

Applications 1. On sensors 2. On robotics

Outline

Previous work Flowchart of the system Algorithms

Current work Improved algorithms Comparisons

Future work Problems unsolved How to enhance computation speed

Previous Work

The original image-based 2D dataset :

Size: 512*512*40*(R, G, B)

Flowchart and Modulus

Input imagesSegmentation

Feature extraction

Classification

Graph building

Basic features

classes

Directed acyclic graph

Previous Work - Algorithms Modulus1: Segmentations

Global Thresholding:

Problems: One threshold to all image sequences. Iterative region growing method [1]

After applying this method to segmented image sequences:

VS

Previous Work-Feature Extraction

Output from Feature extraction module

viewID mx my areas labeling timeID label.

Modulus2: Feature Extraction

After gaining region information from segmentation stage, we can browse each region to find basic features:

Areas – The count of all pixels in the region.Center of Gravity –The center of all points in one region.

Classification /Feature tracking

Modulus3: Classification One Assumption

Time between successive data sets is small: we can assume the difference between a pair of views should not vary too much.

Euclidean Distance Classifier

Aeuc xAxd ,

Current Work-Improvement Modulus 1: Segmentation: Optimal Thresholding: Isodata algorithm

Segment images into two parts using a starting threshold value.

Calculate the mean (mf,0) of the foreground pixels and the mean (mb,0) of background pixels.

A new threshold value is now computed as the average of these two sample means.

The process is repeated, based upon the new threshold, until the threshold value does not change any more.

Previous Work - Algorithms

Modulus1: Segmentations Region growing:

Purpose: Trying to separate overlapped objects

Algorithms: Region growing based on Marr-Hildreth and sobel edge detectors

Current Work-Features extraction

Feature Extraction Diameter - Diameter is the distance between two points on

the boundary of the region whose mutual distance is the maximum.

Major Axis of The Region – the major axis of the region is the line which minimizes:

These two features are relatively robust, and the second feature: major axis can help detect the reflection part on objects.

n

iid

1

22d

Current Work-Features extraction

Feature Extraction Compute major axis

PCA Diameter

1. Rotate the X-Y coordinate to let the new X-coordinate is the major axis

2. Divide the 2D plane into four regions, find the furthest points on each region

3. Calculate the Euclidean distance

)cos()sin(

)sin()cos(Q

Current Work-Features extraction

Experiment results from diameter detector

Current Work-Features extraction

Experiment results of from feature extraction modulus:

TimeID ViewID Mx My R G B areas diameter angle label

Current Work-Feature Extraction

Modulus2: Feature Extraction Problems solving:

Reflections: According to the experiment result on the right:

to some big objects, their reflections which come from the distance transformation when we pre-projected 3D objects onto 2D image plane

are distracted as different objects. Algorithms: Using the property of major axis: because they belong to the same object, their major axis should parallel or at least have similar angles to each other

Current work-Classification /Feature tracking

Classification methods

Euclidean Distance Classifier

Evolution in time-varying images

There are five different changes of regions between a pair of views.

Continuation: one feature continues from dataset at t1 to the next dataset at t2

Creation: new feature appear in t2 Dissipation: one feature weakens and becomes part of the background Bifurcation: one feature in t1 separates into two or more features in t2. Amalgamation: two or more features merge from one time step to the

next.

Current Work

Output from Classification module New class to preserve the output dataset from Classification module:

class LabelTrack(). It preserve the information:1. ViewID: camera positions, we will move camera around the object in

order to restore 3D object.

2. timeID: time order, for each camera position , we will take several time- varying images

3. classID: class number after correspondence computation between a pair of images in time order

4. Label: the original region numbers before correspondence computaton

5. R, G, B: the color information for each pixel

6. Coordinate x, y: the 2D coordinate of the projection of 3D object.

7. Forward pointer: preserve the labeling information of the previous dataset

8. Backward pointer: preserve the labeling information of the next dataset

Future Work-Speed Enhancement

The importance of computation time Size of mine dataset:

512*512*24*40(time orders)*N(camera positions)

In [5], the computation time for 128^3*10 is 7 minutes.

In the previous work, I use 4 minutes for 512*512*24*40.

In the current work, most I/O operations have been removed, although the computation time is around 5 minutes. Most of the time is consumed on Marr-Hildreth edge detector.

REFERENCES

[1] Snyder and Cowart, “An Iterative Approach to Region Growing”, IEEE transaction on PAMI, 1983

[2] Wesley E.Snyder and Hairong Qi, “Machine Vision”, Cambridge [3] Richard O.Duda, Peter Hart, David Stork, “Pattern Classification”, Prenti

ce Hall [4] Rafael Gonzalez, Richard Woods,”Digital Image Processing”, 2nd, Prenti

ce Hall [5] D.Silver, Xin Wang, ”volume tracking”, Visualization '96. Proceedings.27

Oct.-1 Nov. 1996

Thanks

Any questions?