Robot Homing Workshop
Dec 21, 2015
Robot Homing Workshop
1.The Camera Prospective Projection Model
Figure 1.1The camera pinhole model
y'
r
r’
z
y
x
x'(x’,y’)
z
f
Image Planeinverted
Object point(x,y,z)
z
yfy
z
xfx ' '
Figure 1.2Image plane: shifted and rotated
Z
X
Y
x
y r
x’
y’
(x,y,z)
(x’,y’)
Y’
X’
r’
2.Affine Invariant Shape Correspondence Model
),(),...,,(),( 221,1 NN yxyxyxC
Let Shapes C and C’ be described by points lying on their boundaries:
),(),...,,(),( 221,1 NN vuvuvuC
Supposed the following relation exits:
22221
11211
byaxav
byaxau
iii
iii
1100122221
11211
i
i
i
i
y
x
baa
baa
v
u
C’ is an affine transformation version of C (Or vice versa)
For any three points )(),,(),,( , kkjjii yxyxyx from shape C and their
corresponding points )(),,(),,( , kkjjii vuvuvu from shape C’ there exists
the relation:
11110011122221
11211
kji
kji
kji
kji
yyy
xxx
baa
baa
vvv
uuu
111
10011122221
11211
kji
kji
kji
kji
yyy
xxx
baa
baa
vvv
uuu
( Determinant notation )
kjiaffine
kji areaaaaaarea ,,21122211,, )(
1112
1 kji
kji
yyy
xxx
The triangle area created by )(),,(),,( , kkjjii yxyxyx is
)2.1(
2.1 Shape Feature Vector
Let’s look at ),(),...,,(),( 221,1 NN yxyxyxC
For each boundary point Cyx ii )( , we define the feature vector iF
Let ],...,,[ 21T
iMii fffFi Where
111
2
1jiiji
jiiji
yyy
xxx
fji
2
1
N
M
+/-)cyclic around N(
is the triangle area created byfji )(),,(),y ,( ,j-i jijiiiji yxyxx
iF can be groups into 3 parts:
• local features
• global features
• and again local features
)1( )1(
T
......... features features
MiilliikkiliF ffffff
featureslocalglobalocall
i
is normalized to ˆ
1 1
N
i
M
j
i
fji
FiF iF
Now for shape C we can define the set: ˆ ˆ{ , 1, 2,... }C iF F i N
For ),(),...,,(),( 221,1 NN vuvuvuC in a similar way we have:
ˆ ˆ' { ' , 1, 2,... }C iF F i N
According to 2.1
fjiaaaayyy
xxx
aabavvv
uuu
jif jiiji
jiiji
jiiji
jiiji
)(
111
2
1)(
111
2
1' 2121111121211111
, , 11 22 12 21 , , ( ( ) )affinei j k i j karea a a a a area we get
)aa-a(' 21122211 ii FaF
11 22 12 21
11 22 12 211 1 1 1 1 1
' ( )ˆ ˆ ' ' ( )
i i iN M N M N M
i j i j i j
F a a a a F FF i Fi
f ji a a a a fji fji
For each point
,( )i ix y and its affine-transformed corresponding
point ( , )i iu v there exits iFiF 'ˆˆ
2.2 Global Shift Index
Let Shape C’ be an affine-transformed version of C.
),(),...,,(),( 221,1 NN yxyxyxC ),(),...,,(),( 221,1 NN vuvuvuC
If the points from the shape’s boundary have been taken ‘smoothly’ then:
( , ) ( , ) 0,1,..., 1i i i s i sx y u v s N ( + is cyclic around N )
S is the global shift Index.
Estimating s
Estimation of s means finding the most suitable number ˆ ˆ , 0,1,..., 1s s N
to be the value of s.
)ˆˆ()ˆˆ( )()(, isiT
isisi FFGFFE
N
isis EE
1,
G is MxM diagonal matrix1
( )M 2 2
2
2
[ ]
1
2
mm M M
m
mm
G g
g e
)1,...,2,1,0(
minargˆNssEsThe chosen value for s is
2.3 Similarity Matching of ShapesShapes C and C’ are considered similar if they are the same or if one shape is an affine-transformed version of the other.
Let’s define )1,...,2,1,0(
min
NssEE
It is assumed that the feature vector set CF is a unique characteristic of
of a particular shape (or its affine-transformed version)
for similar shapes E is quite small (ideally 0)
for different shapes E should be larger
Local Shift Index
If shapes C and C’ are similar, for each point
)( , ii yx on C
the suitable point )( , iiii ss vu on C’ can be found locally:
)ˆˆ()ˆˆ( )()(, isiT
isisi FFGFFE
)1,...,2,1,0(
,minargNssii Es
Let n be the number of times when
ˆ 0is s n N
For similar shapes n should be close to N.
Similarity Matching Function
For shapes C and C’ ),(),...,,(),( 221,1 NN yxyxyxC
),(),...,,(),( 221,1 NN vuvuvuC
UpxUp
Upxxxh
)(
If
If
ThETotal
ThETotal
C and C’ are similar
C and C’ are different
EEnNh
Total
2
3.Robot Homing Workshop
3.1 Workshop Hardware and Software
Matlab: Getting Image
Matlab: Image Processing
Lejos
vfm
3.2 Workshop Hypothesis• Camera Pinhole Model
Image received by the camera is twisted according to:
z
yfy
z
xfx
• The camera (the robot) is ‘far enough’ from the arena walls
The ratio z
fis approximately constant
The received shape image is an affine-transformed version of the shape printout.
The similarity matching function can be applied to identify the home shape.
3.3 The Matlab Program Aspects
• Traveling around the arena and grabbing the wall images
• Analyzing the received images and retrieving the shape figures
• Determine whether the retrieved shape is the home shape
• Traveling towards the home shape when it has been detected
4 .Analyzing The Camera Images
4.1 Image Segmentation• Converting the RGB Image Into a gray level one.
•Applying Canny Edge Detector with threshold on the converted image.
RGB Image Gray Level Image Applying Edge Detection
• Finding 8-connectivity Image Elements.
1 2 3
8 4
7 56
Pixel’s 8 neighbours
An Image containing two 8-connected elements
8-Connectivity Element Finding Algorithm
1. Scan the image left to right, top to bottom.
2. If pixel value is not 0, then
(a) If only one of its upper and left neighbours (neighbours 1,2,3,8 the previous figure )
has a label, than copy it.
(b) If all neighbours have the same label, then copy the label.
(c) If they have different labels, then copy one of them and enter the labels in
the equivalence table as equivalent labels.
(d) Otherwise assign a new label to this pixel and enter this label in the
equivalence table.
3. If there are more pixels to consider, then go to step 2.
4. Find the lowest label for each equivalent set in the equivalence table.
5. Scan the picture. Replace each label by the lowest label in its equivalent set.
4.2 Analyze of Image Elements
Image Element can be one of the following groups:
1. A whole shape Boundary
Shape Image Suitable Image Element
2 .Part of A shape Boundary
3 .Arena Corners
Image of partial shape & arena corners
Suitable Image Elements
Image of the arena wall
4 .Arena wall
Image Elements due to blank wall image
4.2.1 Wall Detection
• There are “too many” Elements in the image.
4.2.2 Close Line Detection • A whole shape boundary is a smooth close line.(Hamiltonian circle)
We need to find out weather the image element is a closed line .
• The Image Elements due to the arena corners are open lines.
We should find the element boundary.
Image Pixel
Image Interior Pixel
Image Boundary Pixel
Boundary Following Algorithm
.cs
Ss1 .Find the starting pixel for the region using a systematic scan,
say from left to right and from top to bottom of the image .
2 .Let the current pixel in the boundary tracking be denoted by c .
Set cs and let the 4-neighbour to the left of s be .Sb
3 .Let the 8-neighbours of c starting with b in the clockwise order be
821 ,...,, nnn .Find in ,for the first i that is in S.
4 .Set inc and .1 inb
5 .Repeat Steps 3 and 4 until
1 2 3
8 4
7 56
Pixel’s 8 neighbours
• Applying the algorithm to a whole shape boundary (“smooth” closed line)
returns its boundary pixels ordered clockwise.
• If the image element is a smooth closed line the first boundary pixel to be
come across again is only the starting one c.
• This Algorithm doesn’t enable to distinguish between open lines due to
the arena corners and open lines due to partial boundary of a shape
(which may be of interest).
Image ElementsImage
4.2.3 Detection of the Home Shape
• The boundary of the shape considered home is a closed line.
• The (synthetic) home figure is stored in the robot data base.(simply a file)
and its boundary is of course closed. We can use the the matching algorithm described above.
• Sampling in “smooth manner” N points from the home boundary as
in the data base.
• Sampling in “smooth manner” N point from the received shape
boundary (suitable element pixels)
•Applying the matching algorithm
The original home shape
as in the data base
The shape as received by
the robot camera
Problems:
• The received shape is twisted and only approximation of an affine transformation. •The sample points from the received shape can not be promised to be
the corresponding affine transformation version of the sampled points from the
original home shape.
• How to determine NTh , and
Partial Solutions:
• Intuitively the above problems mainly affect the values of the local features and
and have less influence on global features.The matrix reduces the weight of
the local features.
• The values of NTh , and should be found experimentally.
4.2.4 Calculation Performed on a Received Shape.
•Weight of its boundary ( number of pixels)
•The area of the shape bounding box
•The shape’s center of mass
4.2.5 A shape at the image margins• It results in an open line element at the image margins
Elements at the margins should be treated.
Problem: Elements can also be due to the arena corners.
Image
Element in Image
Margins?
Edge Detection
Segmentation
Element Element
Wall?
Closed Shape?
Calculate:Boundary Pixel Number
Bounding Box AreaCenter of Mass
Home?
yes
yes
Image Analysis
5 .Scanning For the Home Shape
• For Traveling inside the arena we need to deduce the 3D orientation out of 2D images.
• Implementing spatial orientation is not an easy task and beyond the scope…
The Scanning process is very limited and primitive.
Detecting the home shape is not promised.
5.1 The Scanning Cycle• The search process is divided into finite number of so called scanning cycles.
• The search is terminated after all the scanning cycles have been exhausted,
regardless the home finding.
StartScan cycle
ClosedShapeFound
HomeFound
WallDetected
Should find ashape OR
Something intop margin
Get an ImageAnalyze it
Move towardshome
Check if the shapeis the smallest
Refine Location
Check if should findshape in next image
Scan cycle ends
Turn towards theSmallest image
Rotate the camerain small angle
Scan cycles exhausted
Move forwards
FinishScanning
yes
no
yes
no
no
no
no
no
yes
yes
Home Scanning Procedure
yes
yes
5.2 The Refine Location Procedure.
The procedure handles two case:
• There is an element at the top margin of the image so it might be a part of a shape
to which the robot camera is to close.
• The current image analysis yields no close shape whereas according to the previous
image a shape should have been found.(element at the left/right margin)
Problem: Because of the insufficient segmentation the elements can be due to
the arena walls,texture and etc.
A finite number of trials should be taken in order to find a shape .
IterNum 1Home Found 0
IterNum<=2and
Element at top margin
Begin
Go backwards
Get an imageAnalyze it
Closed
Shape
Found
Home
Found
IterNumIterNum +1
HomeFound1
yes
yes
yes
no
no
no
Move towards home
End
Refine Location Case 1
HomeFound
ClosedShapeFound
HomeFound1
i = 1
Turn back in small angle
Get an imageAnalyze it
ii +1
Return to theinitial orientation
i<=2
no
no
no
yes
yes
yes
Move towardshome
End
Refine Location Case 2
5.3 Moving towards Home Procedure
• Once the home shape has been detected,the robot moved towards it.
Problem: The home shape doesn’t have to be in the image center so while moving towards it
the shape might be out of the sight field of the camera.
• Calculating the home shape center of mass.
• The robot moves forward in small “steps” checking after each step if the home shape
is still in camera field of sight .
• If not, The robot moves backwards to its previous location and turns a “little” towards
the home shape according to the location of the shape’s center of mass.
Problem: The robot shouldn’t approach the home shape too much,otherwise the
affine transformation approximation won’t be correct to be used and
the matching algorithm won’t be consistent.
• Calculation the area of the home shape bounding box.
• If the area is “too large” the robot stops and doesn’t try to get closer.
End
Begin
Close enough to
home
Move forwards
Get an imageAnalyze it
HomeFound
Move back
Turn back toinitial orientation
End
Turn towardshome
HomeFound
Get an imageAnalyze it
yes
no
yes
no
yes
no
no
yes
Moving towards home
1j
4j
1 jj