1 Final Report On the Implementation of Image Processing Techniques for Target Detecting Robot using OpenCV Project Supervisor Dr. Haroon Attique Babri Submitted By Jahanzeb Nasir 05-Elect-37 M. Usman Saeed 05-Elect-35 M. Mohsin Butt 05-Elect-40 M. Basit Shahab 05-Elect-294 Department Of Electrical Engineering University Of Engineering And Technology Lahore.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Final Report
On the
Implementation of Image Processing Techniques for Target Detecting Robot using OpenCV
Project Supervisor
Dr. Haroon Attique Babri
Submitted By Jahanzeb Nasir 05-Elect-37
M. Usman Saeed 05-Elect-35
M. Mohsin Butt 05-Elect-40
M. Basit Shahab 05-Elect-294
Department Of Electrical Engineering
University Of Engineering And Technology
Lahore.
i
ABSTRACT
Digital Image Processing is one of the many fields that have seen large enhancements in
recent years. Applications ranging from home security cameras to military grade search bots,
cameras to ultra high resolution spectroscopes and what not are some of the application where
DIP is used. The purpose of this project was dual- faceted: educational and also to devise some
method other than what students normally use. MATLAB, with its two immensely popular
toolboxes naming “Data Acquisition Toolbox” and “Image Processing Toolbox” is the
conventional Image Processing tool being used now-a-days. We tried to use some other method
and after a lot of research, we found another tool “OpenCV”. So, in the project we used OpenCV
libraries and functions to perform the Image Processing.
The following report begins with the most basic information about Image Processing
System. The project consisted of three phases. First part was to design a base for the robot and
the interesting part about the base, well actually about the robot is that it was completely made
from wooden scales. The second part was to test the camera on PC and devise the algorithm for
Image Processing. The Image Processing process includes capturing frames, detecting the object
and then telling the robot where the object actually is. The final part was to design the complete
circuitry for the robot and its interface with the PC.
2
TABLE OF CONTENTS
Abstract…………………………………………………………..1
Table of Content………………………………………………....2
List of Figures…………………………………………………....4
Acknowledgements……………………………………………....5
Dedication………...……………………………………………....6
Project Overview…………………………………………………7
1.1 Basic Idea………………………………………………………………...7 1.2 Design Goals……………………………………………………………..7 1.3 Design Parameters for the Robot………………………………………...7
1.4 Design Phases…………………………………………………………….8 1.4.1 Design of Robot structure…………………………………....8
1.4.2 Image Processing……………………………………………..8 1.4.2 Control of the Robot…………………………………………..8
3.2.1.1 Image Sensor in a CCD Camera………………..15 3.2.2 The video Norm……………………………………………...15
3.2.2.1 The Interlace process…………………………...16
3.2.3 Color Camera………………………………………………...16
3.2.3.1 Single chip color cameras……………………….16
3.2.3.2 Three chip color cameras……………………….17
3
\ 3.3 What is Color?...........................................................................................17 3.4 Color management and Color transformation……………………………18
3.5 Color Spaces…………………………………………………………......18 3.5.1 RGB Color space……………………………………………18
3.5.2 Grayscale Color Space……………………………………...19 3.5.2.1 Converting Color Image to Grayscale…………..19 3.5.3 CIEL*a*b* Color space…………………………………….20
3.5.3.1 RGB and CMYK conversions to LAB………….21 3.5.3.2 Range of L*a*b* coordinates……………...….…21
3.6 Steps involved in Digital Image Processing……………………….……21 3.6.1 Capture the image……………………………………….….21 3.6.2 Color space Transformation……………………………......21
3.6.3 Color extraction……………………………………..……...21 \ 3.6.4 Identify an object of specific shape………………………...22
3.7 MATLAB as Image Processing Tool……………..……………………..22 3.7.1 Capture the image………………………………….………22 3.7.2 Color space Transformation………………………………..23
3.7.3 Color extraction…………………………………….………23 3.7.4 Identify an object of specific shape………………………..23
3.8 Open CV as Image Processing Tool………………………………..…...23 3.8.1 Capture the image……………………………………….…24 3.8.2 Color space Transformation……………………………….24
3.8.3 Color extraction……………………………………………24 3.8.4 Identify an object of specific shape……………………….25
Open CV as Image Processing Tool…………………………..26
4.1 Introduction……………………………………………………………..26 4.2 Why Open CV…………………………………………………………..26
4.3 Integrating Open CV with Microsoft Visual C++ 6.0…………………26 4.3.1 Setting up the system PATH environmental variable……….27
4.3.2 Specifying Directories in Visual C++ 6.0………………….27 4.3.3 Setting for new Projects in Microsoft Visual C++ 6.0…….27 4.4 Important Data types and Functions Used ……………………………..27
4.4.1 IPL Image Data Structure……………………………….....28 4.4.2 cvCvtColor………………………………………………..…..30
6.6 Circle Module…………………………………………………………….52 6.7 Draw Module……………………………………………………………..52
Appendix A-Matlab Code……………………………………...53
Appendix B-OpenCV Code…………………………………….56
Appendix C-Datasheets……………………………………...…70
Refrences………………………………………………………...
5
LIST OF FIGURES
Figure Title Page
2.1 Base Design ………………………………………………………….................10 2.2 Arm Design…...……………………………………………………....................11
2.3 Complete Robot Design………………………………………………................12 3.1 Image Processing System………….…………………………………................14 3.2 Interlace Process: two fields make a frame…………………………...…….......16
3.3 Image Produced by one chip camera…………………………............................17 3.4 Incoming light is divided into its basic components……………………………17
3.5 color images & their corresponding grayscale.....………………………………19 3.6 L*a*b* Color Space………………………………………………….................20 4.1 Input Image……………………………………………………………………..42
4.2 Multiple Circle Detection………………………………………………………42 4.3 Red Color Detection……………………………………………………………43
4.4 Blue Color Detection…………………………………………………………...43 4.5 Green Color Detection………………………………………………………….44 4.6 Input Image to Function and Output Image…………………………………….44
5.1 Control of DC Motor……………………………………………………...…….47 5.2 Control of Stepper Motor………………………………………………………47
5.3 Microcontroller Circuit………………………………………………………...48 5.4 Power Supply Unit……………………………………………………………...48 5.5 Serial Port Circuit..……………………………………………………………...49
6.1 Block Diagram of System………………………………………………………51
vi
ACKNOWLEDGEMENTS
We would like to thank all our teachers who taught us at UET because it was just because
of their efforts, we succeeded in achieving our goal.
Special thanks to the advisor, Dr. Haroon Attique Babri for his indispensable advice at
the most crucial steps of the project.
Many thanks to Sir Fahad Zubair, for he was there whenever we needed him. He was
very much helpful in the project and it would be a remiss if we don‟t mention that actually the
whole project was his idea. We wish him Good luck for his future plans.
Last, but not the least, we would like to thank Sir Asif Rehmat for his expertise and
helping us set the direction of the project.
Sincerely,
The Project Team.
7
DEDICATION
We dedicate this project to our Parents and Teachers for it is just because of their
prayers that we were able to complete our goals.
8
PROJECT OVERVIEW
1.1 Basic Idea
The idea behind the project was to develop a robot that can assist in search and
retrieval of a specified object. For that, we decided to set up a game plan for the robot
where there will be balls of different colors and the robot will search and retrieve a ball of
specific color. The searching part was done using a webcam that was interfaced to PC
serially. The retrieval process start once the ball is found.
1.2 Design Goals
Most of the robots that are being built now-a-days have metallic body. That
makes the robot heavy and if not that, expensive and difficult to design. So the first goal
was to come up with a new design and the idea that came up was to use only wooden
scales. The goal was to make the robot strong enough to carry all the motors, battery and
other stuff and on the other hand it should be lighter than the robots having metallic body.
After processing the image on PC, the second goal was to establish a serial link
between the robot and PC that will tell the robot when and where the ball actually is.
1.3 Design Parameters for the Robot
At the kick-off meeting, we decided how the project would be approached. At
this meeting, we decided several general design goals based on what has already been
done on such kind of project. By reviewing the pros and cons of the previous attempts
that were made to construct a robot with Image Processing capability, we finalized our
design goals and started work on finalized design.
The design goals of the robot were to reduce the cost and weight of the robot and
to devise an optimized method for the arm that will pick up the ball. Specific design
parameters are as follows:
2 degrees of freedom (1 for the arm, 1 for the gripper)
One DC motor for rear movement.
9
One bipolar stepper motor for front movement.
A single DC motor for both arm and gripper.
1.4 Design Phases
The whole design was divided in three phases.
1. Design of Robotic Structure.
2. Image Processing.
3. Control of the robot.
1.4.1 Design of Robotic Structure
The very first phase was to design the robotic structure. Like mentioned before,
the structure was made using wooden scales.
1.4.2 Image Processing
In this phase, we used a webcam to capture the image to the PC and then used
different image processing techniques to find the ball in the image. The image processing
tools used were OpenCV and MATLAB.
1.4.3 Control of the robot
This final phase was to implement a control circuitry on the robot which was
interfaced to the PC serially. After the ball is found, the PC sends a signal to the robot
about the location of the ball and then the robot works accordingly.
1.5 Report Contents
The report breakdown is outlined in the table of contents and list of figures. The
report moves forward in exact order of design phases. The first part consists of the
Robotic structure design. After that comes the Digital Image Processing chapter in which
basic concepts of DIP are explained. Also, this chapter includes an overview of our
methodology of performing Image Processing. Image Processing with the help of
MATLAB is explained and the code for frame extraction, color space transformation and
image segmentation is given in the appendix. After that the same procedure is explained
10
using OpenCV. Then comes the main chapter of “Image Processing Using OpenCV” in
which the image processing process is explained step by step. After that the “Design of
Control System” for the robot is explained. The appendices include code for Image
Processing using MATLAB, Image Processing using OpenCV and code for the control
circuit. The appendix also includes future recommendation for the project. The
information presented in the appendices is a supplement to this report, intended to aid
future designers who work on a similar robotic manipulator project (for example, use the
programs for design optimization).
11 | P a g e
Design of Robotic Structure
2.1 Base Design
The first phase of the project was to design of the base and it was rather complex because
we did not have any experience or knowledge of the mechanics involved. Mechanical design of a
robot is an experience unlikely encountered by most Electrical engineering students. The robotic
structure design alone developed into a complete project which pushed us to get some knowhow
of basic mechanics involved. The design of the base is shown in Figure 2.1.
Figure 2.1: Base Design 1
12 | P a g e
The base was designed keeping following factors in mind.
The base should be rigid enough to hold the motors and other stuff aboard.
The size of the base was designed so that it was large enough to accommodate everything
that was needed on board.
The front part was extended to accommodate the webcam.
2.2 Arm Design
The arm design was done keeping in mind that it has to be freely movable. The arm was
supposed to come in action only when the target was in range. In all other cases the arm was
raised above. The basic arm design is shown in figure 2.
Figure 2.2: Arm Design
13 | P a g e
2.3 Complete Robotic Structure Design
The complete robotic structure is shown in figure 2.3. The complete structure consists of
all the motors for front, rear, and arm movement. The front part was extended so that camera can
be accommodated.
Figure 2.3: Complete Robot Design
2.3.1 Rear Motor Assembly
The rear motor is a simple 12 V DC motor placed on a bench on the rear of the robot. The
shaft was made using a screw driver and the tires were custom made using tires available in
market.
14 | P a g e
2.3.2 Front Motor assembly
The Front motor is a unipolar stepper motor with step size of 7.5 degree. The motor was
mounted on the wooden base. The front wheels turning capability is achieved using a rotary to
linear motion gear system.
2.3.3 Arm Motor Assembly
The arm motor is also placed on the base of the robot. It is a simple DC motor. The gears
attached have a very large gear ratio which helps the arm to retain its position and also helps it to
pick up large loads.
2.4 Changes made at the final moment
We had to do an important change in the arm at the final moment. The problem was that
when the designed arm was tested in actual environment, it was unable to hold the ball. So we
had to install some additional hardware for that. A door type arrangement was made. As we had
already accommodated three motors onboard, so we had to devise an arrangement such that only
one motor controls both the arm and the doors. The change made was that we add a pulley on the
top of the robot and the doors were pulled out by elastic bands. The doors were pulled in by a
wire which was pulled by the motor instead. So when the ball is in place and motor starts pulling
up, the doors close first and then the arm is raised.
15 | P a g e
Digital Image Processing
3.1 Image Processing System
An image processing system consists of an illuminating source, an image sensor
(webcam in our case) and interface between image sensor and PC. A simple diagrammatic
representation of an Image processing system is shown in figure 3.1.
Figure 3.1: Image Processing System
The interface between the camera and PC does some very important things. First it
converts the analog video data to digital data so that it can be understood by PC. Then there is a
part of interface known as the frame grabber that grabs the frames at specific instants as they
come from camera. After the frame grabber, the images are available in the drive and can be
processed.
3.2 Image Processing Fundamentals
Image processing is a general term used to describe different techniques used in image
manipulation. Most cameras produce images in form of spatially distributed intensity values of
16 | P a g e
electromagnetic radiations which can be digitized and stored in RAM. This helps in the
processing of the image.
Many image processing systems are used depending on the application used. They differ
in their acquisition principle, speed, resolution, sensor system and range.
Image sensors can be classified according to their sensitivity ranges. Electromagnetic
sensors for gamma radiation, X-rays, visual spectrum, infrared spectrum and radio wave
spectrum are available and are used in different applications.
3.2.1 CCD Cameras
In a film camera, a photo sensitive film is moved in front of the lens, exposed to light,
and then mechanically transposed to be stored in a film roll.
A CCD camera on the other hand has no mechanical parts. Incoming light falls on a CCD
(Charge Coupled Device) sensor which is actually a large number of light sensitive
semiconductors which we call “pixels”.
3.2.1.1 Image Sensor in a CCD Camera
Image sensor is the heart of a CCD camera. The physics behind the sensor is the inner
photo effect. This means that the incoming photons produce electrons in the semi-conductor
material, which are separated in the photo diode and are stored in a capacitor. This capacitor is
connected to the surrounding electrical circuit via a MOS transistor, which acts like a light
switch. If it opens, the charges will be collected by the capacitor (that explains the word
“integrated”) and will be transported when the switch is closed.
3.2.2 The Video Norm
Real time systems are usually based on video norms, which mean that the image
acquisition as well as the conversion of digital data into a video signal and vice versa have to
conform to international standards. In Europe, this norm is defined by the “Comite Consultaif
International des Radiocommunications (CCIR)”; in the USA, the norm is called RS-170
Standard and was defined by the Electronics Industries Association (EIA). The PAL (Phase
Alternation Line) and SECAM (Sequential Couleur a Memoire) color standards are based on
17 | P a g e
CCIR while the color system based on RS-170 is NTSC (National Television System
Committee).
3.2.2.1 The Interlace Process
Both norms require a process which is required for the image on the screen to by non-
flickering. This is called the “Interlace Process”. The complete image (frame) is divided in two
half images (fields). One consists of the odd lines and the other consists of even lines of the
images. The interlace process is shown in figure 3.2
Figure 3.2: Interlace Process: two fields make a frame
3.2.3 Color Cameras
Color cameras produce a color image which consists of three parts: Red, green and blue.
By additive color mixture and intensity variations in the different parts, almost any color can be
produced.
3.2.3.1 Single Chip Color cameras
These cameras are less sophisticated. The incoming light gets separated into its red, green
and blue parts by means of a strip or mosaic filter which is located directly on the CCD sensor.
The image produced by these cameras is shown in figure 3.3. At readout time, the pixels of red,
green and blue are transferred successively. Electronic switches divide the signal into primary
colors. The three primary colors cannot be separated for individual image processing purpose
and so they are normally used in entertainment electronics areas.
18 | P a g e
Figure 3.3: Image Produced by one chip camera
a) Mosaic filter
b) Stripe filter
3.2.3.2 Three chip color cameras
These cameras use a separate CCD sensor for each of the three primary colors. Prisms in
the optical path provide a separation of incoming light into its three components, which will be
directed to the appropriate sensor. The data of three CCD sensors can be stored on RAM and can
be processed separately. The process of color separation is shown in figure 3.4.
Figure 3.4: Incoming light is divided into its basic components
3.3 What Is Color?
Color is an experience. When light reflects off a colored object, the human eye senses
color. Technically, the measure of the power, called the Power Spectral Density (PSD), carried
19 | P a g e
by each frequency or “color” in a light source (illuminant) and the colored object (colorant)
combine to produce reflected light with a particular PSD. This light interacts with the three types
of color-sensitive cones in the human eye to produce nerve impulses that the brain interprets as a
specific color.
Spectral color is useful for unambiguously describing a color, but it can be unwieldy for
describing large numbers of colors. It is much more common to use a color space transformation
that helps in getting device independent colors. In the case of RGB, the received images are
device-dependent combinations of red, green, and blue light. For color systems such as XYZ and
L*a*b* these images become device-independent and are modeled on aspects of the human
visual system.
3.4 Color Management and Color Transformations
Color management is an essential part of the image acquisition, image processing, and
image output workflow. Most cameras give images in RGB or YUV. So the color appearance
and format of pixels change if we change the camera. So we use color transformation to
transform the received color space in a known color space to make the received images device
independent.
3.5 Color Spaces
Some important color spaces like RGB, Grayscale and LAB are described below.
3.5.1 RGB Color Space
An RGB color space can be easily understood by thinking of it as "all possible
colors" that can be made from three colorants for red, green and blue. Imagine, for example,
shining three lights together onto a white wall: one red light, one green light, and one blue light,
each with dimmer switches. If only the red light is on, the wall will look red. If only the green
light is on, the wall will look green. If the red and green lights are on together, the wall will look
yellow. Dim the red light some and the wall will become more of a yellow-green. Dim the green
light instead, and the wall will become more orange. Bringing up the blue light a bit will cause
the orange to become less saturated and more whitish. In all, each setting of the three dimmer
switches will produce a different result, either in color or in brightness or both.