Vision System for Autonomous Navigation Paul Nguyen and Michael Slutskiy ECE 533 Fall 2005 Project
Vision System for Autonomous Navigation Paul Nguyen and Michael Slutskiy
ECE 533 Fall 2005 Project
Table of Contents 1. Introduction................................................................................................................. 3 2. Approach..................................................................................................................... 3
2.1 Key Issues ........................................................................................................... 4 2.1.1 Texture Noise.............................................................................................. 4 2.1.2 Simulated Sandpit ....................................................................................... 4 2.1.3 Missing Barrel............................................................................................. 4
2.2 Plan of Attack ..................................................................................................... 4 3. Work performed .......................................................................................................... 5
3.1 Noise Reduction.................................................................................................. 5 3.1.1 Gaussian Filter ................................................................................................ 5 3.1.2 Median Filter................................................................................................... 5
3.2 Color Segmentation ............................................................................................ 5 3.2.1 “Orange” Filter............................................................................................ 6
4. “Green” Filter.............................................................................................................. 6 4.1 Sobel Filter and Hough Transform ..................................................................... 7
5. Results ......................................................................................................................... 7 6. Discussion ................................................................................................................... 9 7. Conclusion .................................................................................................................. 9 8. References ................................................................................................................... 9 9. Division of Labor ...................................................................................................... 10
2
1. Introduction The purpose of this project is to develop a robust image analysis system that would
allow vision based autonomous navigation and object avoidance. This package will be
part of the control software used on the IEEE Robot Team’s entry into the Intelligent
Ground Vehicle Competition1. Motivated by the fact that humans can easily perform this
task relying entirely on vision, we wish to use digital image processing techniques to
achieve the same capability.
For the competition the vehicle must avoid any obstacles it is faced with (cones,
construction barrels, barriers and potholes) while staying within a lane indicated by solid
or dashed lines. In the past, objects and lines were distinguished from the background
simply by using adaptive threshold segmentation on the blue plane intensity of the
original image. However, this had its shortcomings, particularly if the surface material or
color changed (going from a grassy surface to a simulated sandpit, for example).
Moreover, there was no way to prevent the robot from viewing a gap in the dashed lane
marking as an acceptable path which could result in the vehicle going out of bounds.
The goal is to improve the already existing software to better use visual information
and present more reliable data to the Artificial Intelligence system, which will plan the
course of action for the robot. The scope of this project will not include the AI system,
although much improvement can be achieved there as well. Mainly we will focus on
providing a binary image that is at least as good or better to the current AI
implementation. The current strategy works fairly reliably provided clear path is
represented by black and objects/lines are represented by white in the binary image.
2. Approach Of course, “reliable” and “better” are only useful adjectives when the context is
understood: below are several images that posed problems for the adaptive threshold
system. These are some of the issues we will be attempting to address in this project.
1 Additional information about the competition is available at http://www.igvc.org/deploy
3
2.1 Key Issues
2.1.1 Texture Noise Here the grass texture and
variance in color has created
noise in the binarized image,
which will be interpreted as an
object directly in front of the
vehicle – an undesirable outcome.
2.1.2 Simulated Sandpit Clearly, the thresholding
technique fails us here since the
tarp should not be interpreted as
an obstacle and thus should
appear black in the binarized
image on the right.
2.1.3 Missing Barrel In this image the shadow
side of the construction barrel is
missed and the object for the
most part does not appear in the
final image – unreliable data.
This in fact happened last year.
2.2 Plan of Attack Using these three particular problems as a guide, we decided to implement noise
filtering as well as a color segmentation scheme to aid/replace the threshold
segmentation. Using knowledge gained in class it was determined that the Median and
Gaussian filters may be good candidates for noise removal. Also, we chose to try to
implement several methods that could be helpful in future extensions of the vision system
(for example detecting dashed lane markings or preparing images for mapping). To this
4
end, we wanted to implement (time allowing) the Sobel edge extraction, Hough transform
for line extrapolation and a transform to convert the image into an overhead view.
3. Work performed
3.1 Noise Reduction
3.1.1 Gaussian Filter Gaussian filtering was
unable to eliminate all the noise
in this image. An additional
downside of this filter is that it
tends to blur the image and does
not always preserve edges well.
This was deemed unacceptable, especially when compared with Median Filter.
3.1.2 Median Filter The same image as above is
pictured here after a 5x5 Median
filter and to the right is 5x5
followed by 3x3 filter. This gave
us a great result getting rid of all
noise in the middle of the lane.
However, an unavoidable
downside of the filtering is the
almost full disappearance of the
barrel. Clearly, something had to
be done to reliably find objects
that may be in the shadow.
3.2 Color Segmentation It was observed that much of the information in the image is contained in the color
data. As we were discarding the red and green planes for our threshold technique we were
essentially throwing away valuable knowledge of what lies ahead. We decided to take a
5
two-step approach to extracting this information. First, we would find all orange pixels
(seeing as orange inevitably means obstacle) and turn it to white. Next we would look for
green, brown and yellow hues and change them to black since those most often represent
a clear path. We decided that this “filtering” would be most convenient in HSI color
space. RGB would pose a problem since orange actually contains fairly high levels of
green so there is a potential conflict with the two filters. In HSI, however, this conflict
does not occur since orange is a distinctly different hue. The results are displayed below.
3.2.1 “Orange” Filter The left image is a typical
result of the orange extraction
algorithm. The right image is the
final result after binarizing and
Median filtering. Note how the
barrel is now clearly detected.
Although parts of it are still black, our AI only requires the base of the obstacle to be
clearly visible to function correctly.
4. “Green” Filter
Here are two good examples of cases where the dynamic threshold did not perform
well originally. The middle images are the result of applying the color segmentation
based on green, brown and yellow hue range. And the final images are the result of both
color filters, dynamic thresholding and Median filtering.
Although, the right lane marking is erased in the first example, it is an acceptable
sacrifice – the line would be detected if the robot were to move closer to it, as it stands
however, it is quite far from being a serious threat.
6
While there is a chance that through a better adjustment of the filter it would be
possible to capture the tarp entirely, the median filter makes this almost unnecessary.
4.1 Sobel Filter and Hough Transform As could be seen in several of the above examples, parts of the lane marking are
sometimes eroded by noise filtering and segmentation. Additionally, although we do not
currently have a sample image of this, there are areas of the course where a dashed line
replaces the solid demarcation of the above examples. To deal with both these situations
we sought to implement the Hough transform which would at least partially allow the
robot to “guess” where the lane boundary is based on previous data. Since we already
have a good binary image to work with, it was determined that Canny edge detection was
not necessary and a simple Sobel filter would suffice to remove the “bulk” of the objects.
Although, the Hough transform has not yet been fully integrated into the AI system, we
nonetheless fully completed its implementation. See examples in the results section.
5. Results Below is a comparison of the results of our revamped vision system to the old version.
Original Old Vision System New Vision System Hough Result (blue)
7
Original Old Vision System New Vision System Hough Result (blue)
8
6. Discussion As can be seen in the above images, the new system performs at least as well and in
many cases better than its predecessor in detecting objects, ignoring tarps and eliminating
noise. Therefore, we made good progress in making the vision processing more reliable.
Additionally, we set out with the goal of making our software robust – for vision based
autonomous navigation to be possible the analysis must happen efficiently in real time.
To achieve this we employed the Java Advanced Imaging library (our software platform
is Java based to increase Operating System independence), which employs native
hardware accelerations for imaging operations. This proved to be sufficiently fast in the
past, though it remains to be seen how the new system will perform on “live” data.
Additionally, a 3D Lookup Table was used for our implementation of the Hough
transform to reduce the necessary computations. Further modifications to the code are
being considered to improve efficiency even more.
7. Conclusion This project successfully addressed several known issues with the already existing
vision system on the IEEE Robot Team’s IGVC vehicle by employing filtering and
segmentation techniques learned in class. Additionally, we extended the software
capabilities for future extensions such as mapping by efficiently implementing the Hough
transform. For this purpose we are also looking into adding the ability to synthetically
rotate the images into a top-down view. We expect to have this feature operational in the
near future. Other work will likely focus on fine-tuning the various filters and making the
software as lean as possible to increase reaction time.
8. References 1 Gonzalez, J.P. and Ozguner, U. “Lane detection using histogram-based segmentation and decision trees,”
Intelligent Transportation Systems, 2000. Proceedings. 2000 IEEE, 1-3 Oct. 2000, pp. 346 –351.
2 Yu, B.; Jain, A.K.; “Lane boundary detection using a multiresolution Hough transform,” International Conference on Image Processing Proceedings, 26-29 Oct. 1997, pp. 748 -751 vol.2.
3 Bertozzi, M. and Broggi, A. “Real-time lane and obstacle detection on the GOLD system,” Intelligent Vehicles Symposium, 1996. Proceedings of the 1996 IEEE, 19-20 Sept. 1996, pp. 213 –218.
4 Siegwart and Nourbakhsh. “Cameras, Images, and Low-Level Robot Vision,” Introduction to Autonomous Mobile Robots. MIT Press, April 2004. Sec. 4.1.8.
9
9. Division of Labor Paul Nguyen Michael Slutskiy
Research 50% 50% Program
(Development and Testing) 50% 50%
Presentation 50% 50% Report 50% 50%
Total Contribution 50% 50% __________________________________ __________________________________
Paul Nguyen Michael Slutskiy
10