project report final

Birla Institute of Technology and Science, Pilani 1

A Project Report

On

Coded Target Detection and 3D Coordinate

Points Generation

Name of the Students ID

Sankhya Chakravarty 2012A7PS014P

Aishwarya Gupta 2012A2PS241P

Kunjesh Agashiwala 2012B5A3663P

Ishan Kumar 2012AAPS246H

Vishal Kejriwal 2012A2PS366P

Shubham Singh 2012B3A7466P

Divija Gogineni 2012AAPS003H

Jaiwant Rawat 2012B4A7714P

Prepared in partial fulfillment of the

Practice School – 1

Course No. – BITS F221

At

Indian Institute of Remote Sensing, Dehradun

A Practice School – 1 station of

BIRLA INSTITUTE OF TECHNOLOGY AND SCIENCE, PILANI

23rd

May to 17th

July, 2014


A Project Report

On

Coded Target Detection and 3D Coordinate

Points Generation

Name of the Students ID Discipline

Sankhya Chakravarty 2012A7PS014P (B.E.) Computer Science

Aishwarya Gupta 2012A2PS241P (B.E.) Civil

Kunjesh Agashiwala 2012B5A3663P (M.Sc.) Physics+

(B.E.)Electrical and Electronics

Ishan Kumar 2012AAPS246H (B.E.) Electronics

&Communication

Vishal Kejriwal 2012A2PS366P (B.E.) Civil

Shubham Singh 2012B3A7466P (M.Sc.) Economics + (B.E.)

Computer Science

Divija Gogineni 2012AAPS003H (B.E.) Electronics

&Communication

Jaiwant Rawat 2012B4A7714P (M.Sc.) Maths + (B.E.)

Computer Science

Prepared in partial fulfillment of the

Practice School – 1

Course No. – BITS F221

At

Indian Institute of Remote Sensing, Dehradun

A Practice School – 1 station of

BIRLA INSTITUTE OF TECHNOLOGY AND SCIENCE, PILANI

23rd

May to 17th

July, 2014


Declaration

We hereby declare that the work recorded in this project report entitled “Coded Target

Detection and 3D Coordinate Points Generation” in partial fulfillment for the Practice School –

1 of Birla Institute of Technology and Science, Pilani is a faithful and bonafide project work

carried out at “Indian Institute Of Remote Sensing Dehradun” under the supervision and

guidance of Ms. Shefali Agarwal, Mr. Raghavendra S, Ms. Poonam S Tiwari and Ms. Hina

Pande.

The results of this investigation reported in this project have so far not been reported for

any other Degree / Diploma or any other Technical forum.

The assistance and help received during the course of the investigation have been duly

acknowledged.

Name of the Students ID

Sankhya Chakravarty 2012A7PS014P

Aishwarya Gupta 2012A2PS241P

Kunjesh Agashiwala 2012B5A3663P

Ishan Kumar 2012AAPS246H

Vishal Kejriwal 2012A2PS366P

Shubham Singh 2012B3A7466P

Divija Gogineni 2012AAPS003H

Jaiwant Rawat 2012B4A7714P


Acknowledgement

We take this opportunity to express our gratitude towards everyone who has made it

possible for us to work on this project.

Sincere gratitude and thanks are expressed to Ms. Shefali Agarwal, Mr. Raghavendra S,

Ms. Poonam S Tiwari and Ms. Hina Pande of IIRS Dehradun for sharing their valuable

knowledge about the subject and helping us with their ideas and resource.

We would like to express our thanks to Dr. Pradipta Chattopadhyay, PS-1 Faculty

Coordinator and Mr. Vasant Keshav of BITS Pilani for making the facilities available to us

whenever needed and also giving us encouragement during the course of this project.

We would also like to extend our heartfelt gratitude towards BITS Pilani for allowing us

to proceed with the project and lastly our heartiest thanks to all our friends who helped us in this

project and gave us inspiration and confidence.


Contents

1. Abstract 7

2. Introduction 7

2.1. Theory and Overview 7

2.2. Documents and Research Papers 9

2.3. Organization of the report 10

3. Problem Statement and Analysis 11

3.1. Problem Statement 11

3.2. Detailed Problem Analysis 11

3.3. Solution Strategies 11

3.4. Hardware and Software Requirements 11

4. Algorithm Design 12

4.1. Entire Flow Chart 12

4.2. Step By Step Explanation 13

4.2.1. Image Capturing 13

4.2.2. Grayscale conversion 14

4.2.3. Automatic Code Detection 15

4.2.4. Pattern Matching 17

4.2.5. Internal Camera Calibration 21

4.2.6. Exterior Orientation 22

4.2.7. Generating 3D Coordinate Points 24

5. Implementation 25

5.1. Code 25

5.2. Graphical User Interface 26

6. Conclusion and Future Scope 32

7. Limitations 32

8. References 33


List of Figures and Illustrations

Sl .No Figure Description Page Number

Figure1 3-D Wire-Frame Models 8

Figure2 Hardware and Software

Requirements

11

Figure3 Flowchart of entire Algorithm 12

Figure4 Nikon D-80 Camera used for

capturing images

13

Figure5 Image Capturing Algorithm 13

Figure6 Captured Images 14

Figure7 Flowchart for Grayscale

Conversion

14

Figure8 Grayscale Images 15

Figure9 Flowchart for Dot Detection 16

Figure10 Dot Detected Images 17

Figure11 Failure of Delaunay Triangles 18

Figure12 Possible Position of Codes 19

Figure13 Image showing Rows on the

basis of distance

20

Figure14 Image showing columns of

the basis of distance

20

Figure15 Summary of Internal Camera

Calibration

22

Figure16 Summary of External Camera

Calibration

22

Figure17 Screenshot of the output 24

Figure18 3D Coordinate Points Plotted 25


1. ABSTRACT The project report is entitled “Coded target detection and 3D Coordinate Points Generation”. As

its name goes, the objective of the project is to generate 3 dimensional coordinate points from a

set of 2D images. The entire project is based on the principles of Photogrammetry. The subject of

interest is embedded with patterns of retro-reflective dots. The reason that these dots were

attached is that they appear very bright in the image if it is captured using flash. Several pictures

of that subject were taken at different angles with the same focal length maintaining an overlap

of at least 70%. An auto-bar is used to define the coordinate system reference.

The algorithm which performs this transformation is broken down to several steps. First step is

converting the images to grayscale. Then detecting the retro-reflective dots by the “Polygon”

approach. After detection of the dots in several images, same dots in different images were

matched by Pattern Matching algorithm which is a significant step in the whole process. Then

internal camera calibration was performed which produced the six internal camera parameters

namely focal length, tangential distortion, radial distortion etc. Using this information, the

external camera orientation is found out. Finally, the 3D Coordinate points were generated by the

“Bundle Adjustment” procedure.

2. INTRODUCTION

2.1. Theory and Overview

Photogrammetry is the science of making measurements from photographs, especially for

recovering the exact positions of surface points.

The fundamental principle used by photogrammetry is triangulation. By taking photographs from

at least two different locations, lines of sight can be developed from each camera to points on the

object. These lines of sight are mathematically intersected to produce the 3-dimensional

coordinates of the points of interest.

The output of photogrammetry is typically a map, drawing, measurement, or a 3D model of some

real-world object or scene. Many of the maps we use today are created with photogrammetry and

photographs taken from aircraft. Moreover, it may be used to recover the motion pathways of

designated reference points located on any moving object, on its components and in the

immediately adjacent environment. Photogrammetry may employ high-speed imaging and

remote sensing in order to detect, measure and record complex 2-D and 3-D motion fields.

Its applications include satellite tracking of the relative positioning alterations in all Earth

environments (e.g. tectonic motions etc.), the research on the swimming of fish, of bird or insect

flight, other relative motion processes. The quantitative results of photogrammetry are then used

to guide and match the results of computational models of the natural systems, thus helping to

invalidate or confirm new theories, to design novel vehicles or new methods for predicting

or/and controlling the consequences of earthquakes, tsunamis, any other weather types, or used

to understand the flow of fluids next to solid structures and many other processes.

Types of Photogrammetry


Photogrammetry can be classified a number of ways but one standard method is to split the field

based on camera location during photography. On this basis we have Aerial Photogrammetry,

and Close-Range Photogrammetry.

In Aerial Photogrammetry the camera is mounted in an aircraft and is usually pointed vertically

towards the ground. Multiple overlapping photos of the ground are taken as the aircraft flies

along a flight path. These photos are processed in a stereo-plotter (an instrument that lets an

operator see two photos at once in a stereo view). These photos are also used in automated

processing for Digital Elevation Model (DEM) creation.

In Close-range Photogrammetry the camera is close to the subject and is typically hand-held or

on a tripod. Usually this type of photogrammetry is non-topographic - that is, the output is not

topographic products like terrain models or topographic maps, but instead drawings, 3D models,

measurements and point clouds. Everyday cameras are used to model and measure buildings,

engineering structures, forensic and accident scenes, mines, earth-works, stock-piles,

archaeological artifacts, film sets, etc. This type of photogrammetry (CRP for short) is also

sometimes called Image-Based Modeling.

Industrial Photogrammetry

Industrial Photogrammetry is a sub discipline of photogrammetry. To be very precise, it is

synonymous to Close-range Photogrammetry. But precision is of supreme importance in

Industrial Photogrammetry. It is an accurate, cost effective technique of collecting measurements

of a real world object or environment, directly from photographs. It adopts the theories and

methods of Computer technology, the digital image processing, image matching, pattern

recognition, etc. In recent years, close-range photogrammetry has developed quickly and has

become an effective and precise measuring tool for industrial metrology. 3D Modeling has

become extremely popular nowadays. There can be many ways to represent a 3D model which

include wireframe modeling, 3D point cloud generation and many more. Some 3D wireframe

models are shown below. These reveal how powerful these models can be for exploring the inner

finer structures of any object, howsoever complicated might it be.

Figure1:- 3-D Wire-frame models

Our project, though codenamed “Coded Target Detection and 3D Modeling” does not actually

produce 3D Wireframe model or generate 3D point cloud. Instead, it finds out the 3D coordinate

points of the retro-reflective dots attached to the subject of interest.


A retro-reflective dot appears very bright in the image if it is captured with flash. Actually, the

concerned object is embedded with patterns of retro-reflective dots. Some of these patterns are

shown below

2.2. Documents and Research Papers

For completion of our project, we had an extensive study of research papers and various online

as well as offline documents. Some of these helped us very much in this project and need special

mentioning.

Research Papers

Automatic Camera Calibration in Close Range Photogrammetry by Clive S. Fraser

(University of Melbourne) – This Paper discusses the impact of automation which has

greatly simplified the calibration tasks and the functional model adopted for self

calibration. Issues discussed include interior orientation stability, calibration reliability,

focal plane distortion, image point distribution, variation in lens distortion with image

scale, etc.

Automatic Detection and Decoding of Photogrammetric Coded Targets by Udaya

Wijenayake, Sung-In Choi and Soon-Yong Park (School of Computer Science

Engineering, Kyungpook National University, South Korea)- this paper introduces an

automated coded target detection method which can be used to enhance the

efficiency of the Photogrammetry.

An Implementation of Camera Calibration Algorithms by Meredith Drennan

(Department of Electrical and Computer Engineering, Clemson University) - This paper

seeks to provide an introduction to camera calibration procedures. It also discusses an

implementation of automating point correspondences in known planar objects. Finally,

results of a new implementation of Zhang‟s calibration procedure are compared to other

open source implementation results.

Online Documents

MATLAB® Creating Graphical User Interfaces (R2014a) by Math works. (The

MathWorks, Inc. 3 Apple Hill Drive Natick, MA)

Offline Documents

Building a Matlab GUI by Todd Wittman

Graphics and GUIs with MATLAB®(3rd

Edition) by Patrick Marchand and O. Thomas

Holland

A Guide to MATLAB for beginners and Experienced Users By Brian R. Hunt, Ronald L.

Lipsman and Jonathan M. Rosenberg

Introduction on Photogrammetry by Koert Sijmons (ITC, Netherlands)


2.3. Organization of the report

The entire report has been divided into several logical sections.

Initially, Abstract gives brief information about the project summarizing its main aspects.

The Second section starts with Introduction which includes information about the basic

principles of Photogrammetry with special mention to Close Range Photogrammetry. The

various research papers and documents which were referred for the project are also listed.

The Third section consists of Problem Statement and Detailed Problem Analysis. The

Proposed methodology is mentioned. The Hardware and Software requirements for our

software bundle to run are written.

The Fourth section is divided into 7 sub-sections namely Image Capturing, Grayscale

conversion, Automatic Code Detection, Pattern Matching, Internal Camera Calibration,

Exterior Orientation and generating 3D Coordinate Points. The Image capturing instructs

the user on how to capture the images. As the name suggests Grayscale Conversion deals

with the conversion of images to grayscale. In Automatic Code Detection, the process of

how the retro-reflective dots are identified is described in detail. The Pattern Matching

deals with the automatic matching of the retro-reflective dots which is further used in the

generation of the 3D coordinate points. In Camera Calibration, the intrinsic parameters of

the camera such as focal length, radial distortion, angular distortion, scale factor and

principal point are computed. With the help of these parameters, the Exterior orientation

of the camera is calculated.

The Fifth section covers the implementation details which provide the final design step of

the module. The various tables formed are also provided.

The sixth and Seventh section comprises of the results and conclusions that are drawn

from the algorithm so framed. This is the final output of the process.

Concluding all this will be the Summary and Limitations section which will round up the

entire discussion. It includes summary of achievements, main difficulties encountered

and how they were tackled, and future scope of the project.

This will be followed by the reference/bibliography section that will tell about the books,

papers and sites that have been referred by us while developing the model


3. PROBLEM STATEMENT AND ANALYSIS

3.1. Problem Statement

The problem is to generate 3D coordinate points of any Industrial Object. By Industrial Object,

we mean man-made objects ranging from simple ones like box, glass, cones, cylinders etc. to

very complicated ones like the jet engine architecture of a Boeing all of which have regular

geometric shapes. The subject is embedded with retro-reflective dots. A set of two dimensional

photographs of the concerned subject are given.

3.2. Detailed Problem Analysis and Solution Strategy

A set of 2D images of any object embedded with retro-reflective in them were given to generate

3D model it so as to ultimately match it with its ideal model, i.e. its intended design, so as to

remove any glitches/defects in the real-time object.

Following steps are to be followed in order to achieve the objective-

1. Conversion of images to grayscale so as to reduce the processing size and increase the

contrast between the image and the targets.

2. Detection of dots using polygon approach in order to accurately determine their shape

in any orientation.

3. Camera calibration to compute internal camera parameters which are to be used later

for exterior orientation.

4. Pattern matching among coded targets of various images as per their orientation,

using concepts of slope and distance.

5. Computation of exterior or spatial orientation of the camera.

6. Generation of the 3D model using the above two steps.

Accuracy to the sub-micron level is to be achieved

3.3. Hardware and Software requirements

The Software Bundle that we developed requires the following features to run smoothly on a

computer.

Minimum Requirement Recommended

Hardware

Intel Pentium Processor

1GB of RAM

1GB of free Hard Disk Space

Intel core i5 or above

4GB of RAM or more

2GB of free Hard Disk Space or more

Software

Windows XP ( We strongly

recommend not to use Windows XP on

your Personal Computer as Microsoft

stopped providing support for this

Operating System.)

Windows 7 or 8


MATLAB 2012a MATLAB 2013a

Fig2:- Hardware and Software Requirements

4. ALGORITHM DESIGN

4.1. The Flow Chart

The entire algorithm can be broken down into several steps as written below.

Capturing the images of the subject. The subject is embedded with patterns of retro-

reflective dots. The images thus captured are in the RGB .jpeg format.

Converting the RGB images to Grayscale images.

Detecting the retro-reflective dots in the image.

Matching the dot patterns across the different images.

Calibrating the camera so as to find the internal camera parameters.

Finding the External camera orientation.

Using Photogrammetric Equations, finding out the 3D coordinates of the concerned

subject.

The workflow can be represented by the following flow chart.

Fig3:- Flowchart of the entire Algorithm

Start image capturing converting to

grayscale

detection of retro reflective dots by polygon approach

automated pattern matching

camera calibration to find out the

internal parameters

using photogrammetric equations to find

the 3D coordinates

to plot 3D coordinate points

end


4.2. Step by Step explanation

4.2.1. Capturing the images

The first step is to capture the images of the subject. Nikon D-80 Digital Single Lens Reflecting

camera was used to capture the images. The subject of interest was a cardboard box. It was

embedded with retro-reflective dots. A retro-reflective dot appears very bright in the image when

it is captured using flash. Various images were captured at differing angles.

However while taking the images; the user must keep three points in mind.

The subject of interest should not be captured in much tilted positions as it will lead to erroneous

results.

The images must be taken using the flash, otherwise the retro-reflective dots won‟t appear too

bright to be detected.

All the images must be captured at fixed focal length.

Fig4:-Nikon D-80 DSLR which is used to take the images

The process of taking the images can be shown in the following flowchart.

Fig5:- Image Capturing Algorithm

The captured images looked like this.

The concerned subject is embedded with retro reflective dots.

Images were captured by Nikon D-80 DSLR at various angles using flash keeping the same focal length.

The retro reflective dots appeared very bright as compared to other parts of the image.


Fig6. Captured images

4.2.2. Grayscale Conversion

The captured images were initially in RGB. Before any further processing, it is necessary to convert these

into grayscale. The standard MATLAB function RGB2GRAY was used to convert the images.

Fig7:-Flowchart for Grayscale conversion

After conversion to grayscale, the images looked like this.

RGB image

Using standard MATLAB function RGB2GRAY

Grayscale image


Fig8. Grayscale images

4.2.3. Dot Detection

After the grayscale conversion is over, the process of detecting the dots started. For detecting the

retro-reflective dots, the Polygon approach is used.

Why this approach? The images were captured at varying angles and the retro reflective dots were appearing in

various shapes (mostly ellipse) and sizes. As a result, these dots were not exactly circular and the

standard command IMFINDCIRCLES could not detect the centers accurately. Moreover, the

edges of the dots were not sharp enough.

The polygon approach can be broken down to several steps which are,

First, we used IMFINDCIRCLES to estimate the approximate center and radius of the

dots.

Then, a concentric circle with a greater radius is plotted such that the entire shape is

inside the outer circle.

The entire outer circle was searched for the white pixels which were stored in a matrix.

The standard CONVHULL function of MATLAB is used to recognize the exact

polygonal shape of the dot.


The polygon was plotted along with the images.

After that, the centroids of the polygons were found out.

We can represent the entire algorithm in a flowchart.

Fig9:-Flowchart illustrating dot detection

The dots that were detected in the images looked like this.

• Used IMFINDCIRCLES to detect the approximate centers & radius of the dots.

Finding circles

•Plotted a concentric circle with a greater radius using VISCIRCLES

•Searched for all the white pixels within the outer circle

Reducing inaccuracy

•Found the Polygon using CONVHULL function

Finding Polygon


Fig10. Dot detected images

4.2.4. Pattern Matching

Once the dots were detected it was now required to identify the centroid of the dots of the coded

targets. For this the following formula was been used.

The centroid of a non-self-intersecting closed polygon defined by n vertices (x0, y0 ), (x1,y1), ...,

(xn−1,yn−1) is the point (Cx, Cy), where

After the centroids were detected different algorithms and methods were being worked upon to

match the set of centroids in each coded target with the corresponding one in another image.

Previous approach

1) Delaunay Triangulation


It is used to create a 2-D or 3-D triangulation from a set of points. In a 2-D Delaunay

triangulation, the circumcircle associated with each triangle does not contain any points

in its interior.

Though it is one of the finest methods but it didn‟t work in our images i.e. the Delaunay

Pattern was not same in two of the images which are shown below: (the diagonal of the

square is different in both the images)

Fig11:- Failure of Delaunay Triangulation

Our Approach

Detection of 8 points in order :-

1) First the center of coded target was found using distance

2) After finding the center the main diagonal was isolated from all the present diagonal in the

coded target

3) After this the two symmetric points were found out

4 ) After detecting the above 5 mentioned points in the coded target the main 3 points were

detected .

5 ) After detecting the rest three points the were stored in an array in a sorted manner on the

basis of x - coordinate

6) After this all the 8 points of coded target were stored in an array in the following order

1 -> first point of diagonal having minimum distance from center ( YELLOW )

2 -> point which was right to the firt point ( PURPLE )

3 -> point which was left to the firt point ( BLACK )

4 -> center of coded target ( RED )

5 -> second point of diagonal ( BLUE )

6 , 7 , 8 - > points sorted on x coordinate basis

6 ( CYAN ) 7 ( WHITE ) 8 ( GREEN )


Detection of autobar

1) First the 6 points of autobar were found out

2) After this the found points which were lying on line were isolated

3) After this the two symmetric points were stored on the basis of rotation of the autobar

COLORING OF AUTOBAR :

1 - > WHITE

2 -> RED

3 - > CYAN

4 - > YEllOW

5 -> BLACK

6 ->GREEN

Segmentation of Circular Dots in Coded Targets

This method is based on the simple concepts of ratios and distances.

Fig12:- Possible positions of codes

The entire methodology can be broken down in the following steps:-

First on the basis of slopes and ratios, new coordinates axis – x-axis, y-axis and the

diagonal axis are plotted.

Then on the basis of ratios of the distances of the various points rows are marked as

following.

0

5

10

15

20

25

30

35

0 10 20 30 40

Series1


Fig13:- Image showing the rows on the basis of distances

Using distance formula, perpendicular distances of the three points are calculated from

these row lines.

The row line to which the points are nearest is the corresponding row of the points.

For the allotted rows, ratios of columns were used and the absolute differences of the

distance of these unknown points were obtained. The one which was the smallest was

taken.

Fig14:- Image showing the columns and the differences of the distances

Matching of Coded Targets

• Each image has n number of coded targets. All the center coordinates of the one target

are stored in a file.


• Each coded target has 8 center coordinates. 4 of them form a square and 5th one is on the

diagonal.

• The remaining three can lie on either side of the diagonal. There can be at most two

targets on either side of the diagonal.

• If a side of a diagonal contains one point, then the other point on that side of the diagonal

is given value -1.

• The row and column of the 4 positions, two on either side of the diagonal, are calculated

for each coded target and stored in a file.

Storing the set number of Coded Targets

• First those sets are chosen in which 8 points are completely matched. Then in next round

of matching, the remaining sets are matched to the most suitable one.

• If there are less than 5 points matched for a particular set, it will show the message for

image to be discarded or user to choose the rows and column for that particular set.

Storing the Center Coordinates

• Center Coordinates are extracted from their respective center coordinates file and stored

in a file.

4.2.5. Internal Camera Calibration

This entire process finds out the five internal camera parameters namely focal length, scale

factor, radial distortion, angular distortion and principal point.

A calibration procedure for accurately determining the pose and internal parameters of several

cameras is described. Multiple simultaneously-captured sets of images of a calibration object in

different poses are used by the calibration procedure. Coded target patterns, which serve as

control points, are distributed over the surface of the calibration object. The observed positions

of these targets within the images can be automatically determined by means of code band

patterns. The positions of the targets across the multiple images are then used to infer the camera

parameters.

The code for camera calibration algorithm was originally created by Janne Heikkila, University of

Oulu, Finland.

This program takes camera specifications, a matrix consisting of 3D coordinates, image

coordinates and the normal vector of control points and outputs the intrinsic camera parameters

which are focal length, radial distortion, principle coordinates, angular distortion and scale factor.

The process can be shown in a flowchart.


Fig15:-Summary of Internal Camera Calibration

The input matrix has nx8 data elements which serve as the image co-ordinates of the control points.

Initially the code was made to run on two images which gave the focal length of about 19.37. As we

increase the number of images, we can get more accurate results with standard error in pixels of about

0.000278. The rough elapsed time is about .1 seconds.

4.2.6. Exterior Camera Orientation

As the name suggests, this process finds out the camera orientation. Camera orientation is

nothing but it‟s 3D spatial coordinates along with three spatial angles totaling six parameters.

The entire algorithm can be shown in the following smart-art.

Fig16:- Summary of External Camera Calibration

The exterior parameters of the camera are the three dimensional co-ordinates of the camera and

also the angular orientation of the camera while the image is being captured. The main objective

of this section is to establish a relationship between the digital image(pixel) co-ordinate system

and the real world (latitude and longitude) co-ordinate system. For achieving this, the

collinearity equations were used.

The collinearity equations are a set of two equations used in photogrammetry and remote

sensing used to determine the object coordinates( three dimensions, namely the X, Y, Z of the

1. Camera Properties like Focal length, CCD Chip size, Pixels in Horizontal & vertical Direction

2. Image Matrix containing spatial coordinates, image coordinates and normal vector of the control points.

Inputs The internal camera parameters

1. Focal length

2. Angular distortion

3. Radial distortion

4. Scale factor

5. Principal point

Outputs

The internal camera parameters

1. Focal length

2. Angular distortion

3. Radial distortion

4. Scale factor

5. Principal point

input 3D spatial

coordinates of the subject.

output


object) using the coordinates in the sensor plane.( two dimensions). The collinearity equations

are given as follows:

In the above Equation the parameters involved are :

Rij (i=1,2,3 ; j=1,2,3) are the elements of the rotation matrix.

( X, Y , Z ) are the 3-D global coordinates of the object

x, y are the image co-ordinates of the object.

c is a constant which is determined initially.

The above equations describes the transformation from the object space (X,Y,Z) to the image

coordinates (x , y) . For implementing the process in MATLAB, codes were obtained from the

website of ITC (University of Twente) , Netherlands. The codes obtained were based on the

two basic principles:

1. Newton's Method: It is an iterative method which is used for determining the root of any

given real polynomial or any real transcendental function. It is mainly used in numerical

analysis, and also finds extensive applications in other fields.

2. Least Square Method: The method of Least squares is a standard approach to

approximate solutions of over determined systems, that is, systems in which there are

more equations than the unknown parameters. It is a method which is extensively used

in regression analysis, and is one of the fundamental principles in determining the

exterior parameters of the camera ,which then help in determining the 3-D co-ordinates of

any desired point of the object.

How the Code works:

The code used is used to obtain the spatial orientation of the camera, while the image is

being captured. Initially, the input parameters of the code mainly include the focal length

of the camera, which is obtained from the internal camera calibration process. Also, a set

of random values of the exterior parameters ( Approximate values obtained by trial and

error method) are used as input arguments. The output consists of the adjusted exterior

parameters of the camera ( which are the correct parameters) and also a graph showing

the correction of orientation angles vs. number of iterations to reach the final output.

The code mainly uses the Newton's Method along with the Least Square Method to

iterate the initially obtained values to calculate the final adjusted exterior parameters,

which are the correct parameters.

A screenshot showing the final output of the exterior parameters is shown as follows:


Fig17:- The screenshot of the output of the exterior parameters section.

Thus, from the above graph we can clearly see that the exterior parameters section help us in

determining the correct spatial orientation of the camera which is then further used in

determining the 3-D co-ordinates of the coded targets.

4.2.7. Generating 3D Coordinate Points

• Bundle Adjustment

The Bundle Adjustment is the program that processes the photographic measurements to

produce the final XYZ coordinates of all the measured points. In order to do this, it must

triangulate the target points, resect the pictures and Self-calibrate the camera.

The real power of the bundle adjustment is that it is able to do all three of these things

simultaneously. If you review the descriptions of Triangulation and Resection, it appears there is

a problem. In order to triangulate the measured points, we must know the orientation of the

pictures.

However, in order to orient the pictures, we must know the coordinates of the measured points.

How do we get started here- The answer is the bundle adjustment has the capability to figure

them both out simultaneously and to self-calibrate the camera as well! This is where the name

bundle adjustment comes from because it bundles all these things together and solves them all at

the same time.

• How the code works:

The code used is to obtain the 3D metric coordinates of autobar and coded target points. This 3D

model generation links with all the pervious stages of the project. It takes the outputs of pattern

matching, camera calibration, exterior orientation as inputs. It works on the principle of

Newton‟s method (to solve collinearity equations) and least square adjustment method.

The camera positions of two images(x,y,z,w,p,k) which are obtained from exterior orientation,

focal length (obtained from camera calibration), autobar and coded target points in image

coordinate system (obtained from pattern matching) are given as inputs. The code is developed


in such a way that the output can be given for two images. In order to increase the accuracy,

combinations of n images taken twice are fed into the code and the average is taken.

.

Fig18: 3D coordinate points plotted

5. IMPLEMENTATION 5.1. Code Summary

Convert to Grayscale:

1. Input: The original RGB image

2. Output: The image in the grayscale form

3. Description: The above function converts the original image into the grayscale

form.

Dot Detection;

1. Input: The image in the grayscale form.

2. Output: The grayscale image with all the retro-reflective dots being detected and

their centers marked.

3. Description: The 'Dot Detection' function detects all the retro-reflective dots in the

grayscale image which is being given as input within a particular radius range.

The Autobar is also detected and separately marked.

Camera Calibration:

1. Input: The image with the Autobar and the dots detected in them.

2. Output: The internal camera parameters which includes the focal length, scale

factor, angular distortion etc.

3. Description: The above function calculates the internal camera parameters using

the centers of the Autobar which were initially detected.

Pattern Matching:

1. Input: The image with the centers of the coded targets and the Autobar being

detected.


2. Output: A text file named 'finalcord.txt' with the retro-reflective dots in all the

images being matched.

3. Description: The Function matches the retro-reflective dots in all the images

which have been initially given as input.

Exterior Orientation:

1. Input: The internal camera parameters along the Autobar co-ordinates.

2. Output: The external parameters of the camera which includes the spatial and

angular orientation of the camera.

3. Description: The function calculates the external camera parameters of the

camera.

3-D co-ordinate Points:

1. Input: The focal length, the external parameters along with the matched coded

targets.

2. Output: The figure showing the 3-D co-ordinate points of all the retro-reflective

dots.

3. Description: The function calculates and plots the 3-D co-ordinate points of all the

retro-reflective dots.

5.2. GUI

The software generates the 3-D coordinates of the object from a given set of its 2-D images.

Follow the steps mentioned below-

1. Click some images of the subject.

Note that the subject must be embedded with the Retro-Reflective targets. The

images must be captured with the flash on.

Images should be clicked from a distance of about 1m.

Beware not to click the images in much tilted positions.

2. Open the GUI and click on „Run‟ button ( ) or press (Ctrl+T).

A popup box appears with options to create a new project or open an existing

project or get help or access the web.


If „Create a New Project‟ option is selected, enter the name of the project you

want to create. A new folder of the given name is created in the specified

directory.

All the files henceforth generated by the software will be saved in the same

folder.

3. Load the images

Enter the number of images, to be processed, in the edit box.

Click on „Import Image‟ button. A „Select Image‟ Window appears. Select the

images from any directory.


Convert the images to Grayscale using the Convert to Grayscale Button.

Arrow 1 shows the list of original images.

Arrow 2 shows the list of grayscale images.

Images appended in these list boxes can be viewed individually on the screen by selecting them.

Also, the „.txt‟ files which store the center coordinates of the coded targets of the images can be

viewed from here.

4. Detect the Retro-Reflective Dots using „Dot Detection„ Button.

You need to enter the minimum and maximum possible radius of the Retro-

Reflective Dots as appearing in your images. The maximum radius should be less

than 3 times the minimum radius.

1

2


Centers of the coded targets are stored in the matrix which can be viewed as soon as dot

detection is completed for all the images or by selecting the matrix of any particular image from

the list box as indicated by the arrow in the above image

5. Click on „Camera Calibration‟ button.

The first matrix that appears on the screen gives the image matrix, indicated by

Arrow 1.


The pop up box that appears next gives internal parameters of the camera,

indicated by Arrow 2. These along with exterior camera parameters are used to

generate 3D model.

6. Click on “Pattern Matching‟ button.

It matches coded targets in different locations of the images so loaded.

2

1


7. Click on „Exterior Orientation‟ button.

These graphs show correction of orientation angles with the number of iterations.

Least squares method is followed.

The popup box shows external camera parameters.

9. Click on „3D Coordinates’ button.

All the 3D coordinates will be shown in a pop up window.

User can rotate as he needs with the rotation tool.


6.CONCLUSIONS AND FUTURE SCOPE

FUTURE SCOPE-

• The final co-ordinates of the coded targets can be found out by using a 3-D conformal

transformation.

• For obtaining the 3-D model, the co-ordinates obtained in the output are plotted and the

points are densified to generate the required 3-D model.

7. LIMITATIONS There are some minor limitations of the software developed during the project.

The subject of interest must not be captured in much tilted positions (at the minimum

angle of 60 degrees with the plane in which the object is kept). The dot detection

algorithm will otherwise detect more than one circle for the same coded target. In some

cases, it might not detect any circles for some coded targets. Hence, rejection of such

images should be done manually.

The image size should be no less than 3872 X 2592 pixels in dimensions. Otherwise the

dots would not be sharp enough to be recognized by the software. However it depends on

the size of the retro-reflective dots attached in the subject and the way these appear in the

images.

The user should input minimum and maximum possible radius of the retro-reflective dots

as appearing in the image in pixels manually. User also needs to enter segmentation

distance in pixels which is the average distance between two coded patterns.

1 • The basic principles of Photogrammetry

2 • Camera Calibration

3 • Exterior Orientation of Camera

4 • 3D Coordinate Generation

5 • How to build a MATLAB GUI


8. REFERENCES

Online

http://en.wikipedia.org/wiki/Photogrammetry

http://www.photogrammetry.com/

http://www.geodetic.com/v-stars/what-is-photogrammetry.aspx

http://en.wikipedia.org/wiki/Camera_resectioning

http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/parameters.html

http://www.umiacs.umd.edu/~ramani/cmsc828d/lecture9.pdf

http://en.wikipedia.org/wiki/Direct_linear_transformation

http://www.vision.caltech.edu/bouguetj/calib_doc/papers/heikkila97.pdf

http://cronos.rutgers.edu/~meer/TEACHTOO/PAPERS/zhang.pdf

Introduction on Photogrammetry-Koert Sijmons.

www.wikipedia.org/wiki/Newton‟s_method

www.wikipedia.org/wiki/Least_squares




www.opticalengineering.spiedigitallibrary.org

www.crcsi.com.au

www.geodesic.com/v-stars/what-is-photogrammetry.aspx




http://en.wikipedia.org/wiki/Camera_resectioning

http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/parameters.html

http://www.umiacs.umd.edu/~ramani/cmsc828d/lecture9.pdf

http://en.wikipedia.org/wiki/Direct_linear_transformation

http://www.vision.caltech.edu/bouguetj/calib_doc/papers/heikkila97.pdf

http://cronos.rutgers.edu/~meer/TEACHTOO/PAPERS/zhang.pdf

www.wikipedia.org/wiki/Newton’s_method

www.wikipedia.org/wiki/Least_squares




www.opticalengineering.spiedigitallibrary.org%20

www.crcsi.com.au%20

www.geodesic.com/v-stars/what-is-photogrammetry.aspx

project report final

Documents

practice school

project report

bonafide project work

vasant keshav of bits

coded target detection

d coordinate points

partial fulfillment

raghavendra s