Law cost portable machine vision system

1

Visual Based Product Identification System

by

W.F.R.Madushanka (E/11/252)

M.S.P.Muthukumaranage(E/11/267)

Supervised by Mr.Mahanama Darmawardhana

Interim Report

Submitted in partial fulfillment of the requirements

for course unit PR 406: Industrial Assignment

May 31, 2016

Department of Production Engineering

Faculty of Engineering

University of Peradeniya

1

Declaration

Declaration We hereby declare that this project report is our original work and is written by us in its

entirety. We have duly acknowledged all the sources of information including all images and tables

which have been used in this report

. …..........................................

M.S.P.Muthukumaranage

…...........................................

W.F.R.Madushanka

2

Acknowledgements

This report carried out the contain of our project Vision based product identification system for the PR

406 cause industrial assignment for the completion of degree of BSc Engineering, University of

Peradeniya.

First of all, our sincere thank goes to the supervisor of the project Mr.MahanamaDarmawardhana

Faculty of Engineering, University of Peradeniya for providing us this opportunity to gain a good

knowledge and experience in project.

We also grateful to everyone who supported us for the successful completion of our project other staff

members of engineering workshop and production department.

M.S.P Muthukumaranage

W.F.R Madusanka

Faculty of Engineering,

University of Peradeniya,

Peradeniya.

3

Executive Summary

The eye is one of the most important organs of the human body. So our skills greatly depend on our

capability to see, detect and categorize objects, and to estimate distances. Most of the industrial jobs

centralized on our ability of visual perception. Today machine vision has improved in production

technologies more and more often extend well beyond the limits of human visual capacities. This is

where machine vision technology comes in.

Our project is on vision based product identification system. In this project machine able to identify

objects based on colour and shape. Our aim is to build a system that can detect, recognize objects

according to the colour shape and barcodes. The whole process should be done in real-time, the thing

that necessities employing fast and efficient algorithms. In addition, the actual implementation of our

system should take practicality and ease of use into account. This is because a system as ours is

intended to be used in daily life and hence needs to be both simple and efficient to use.

There are so many methods to solve this problem. In our project single board computer use OpenCV

and Python as software. OpenCV is open source computer vision library. OpenCV and Python is one

of the best combinations. Instead of personal computer we use raspberry pi minicomputer.

In our project we successfully completed and implemented a vision system that can move objects

through conveyor and detect shape colour in real time.

4

Table of Content

1. Acknowledgement 2

2. Executive Summary 3

Chapter 1 Introduction

1.1 Overview 6

1.2 Problem Description 7

1.3 Software and hardware 7

Chapter 2 Solution Methodology 12

2.1 Install software to raspberry-pi mini computer 13

2.2 Colour detection method 14

2.3 Shape detection method 14

2.4 Process flow chart 15

2.5 Project layout 16

2.6 Limitations 16

Chapter 3 Results

3.1 Colour detection results 17

3.2 Shape detection results 18

Chapter 4 Deliverables 20

Chapter 5 Conclusion 21

References 22

Appendix

5

List of figures

1. Fig 1. Raspberry pi minicomputer 7

2. Fig 2. Light room. 9

3. Fig 3. Ring light. 9

4. Fig 4. Spot light. 9

5. Fig 5. Conveyor. 10

6. Fig 6. RGB colour wheel 13

7. Fig 7. Relationship between hue and saturation 13

8. Fig 8. Relationship among hue saturation value 14

9. Fig 9. Project layout 16

10. Fig 10. Red colour identification 17

11. Fig 11. Bluecolour identification 17

12. Fig 12. Square identification 18

13. Fig 13. Triangular identification 18

14. Fig 14. Circle identification 19

6

CHAPTER 1 – Introduction

1.1 Over view

With the rapid development of the production and manufacturing world demands on production

quality and production rate should be increased. For that instead of human eye and brain, have to use

another vision system. According to the above necessity various vision based product identification

system came to the manufacturing world.

Vision based product identification system is considered with the use of image processing technique.

Image processing is a method of analyzing and manipulating images with a computer or can be define

as capture real image and extract useful information from it. In the image processing input and output

both are images. Basically there are three steps include in image processing.

1. Importing the image via image acquisition

2. Analyzing and manipulating the image

3. Extract useful information and show output image with that information.

There are two types of image processing methods. Analogue image processing and digital image

processing. In the past most of image processing was done by using analogue image processing but

with rapidly development of the computer technology digital image processing overcome analogue

image processing.

There are many applications of vision based product identification system. For automotive visual

inspection system is the one of the major application. It is help to improve product quality and

production rate. In the automotive visual inspection system, we can have identified various kinds of

product, defective product, etc. Therefore, this project is the most helpful to production world.

Vision based product identification system can identify various characteristics. In our project Problem

is detecting objectives according to the colour and shape and send unique data for each object. The

project can be divided into mainly two sections. The first section is image capturing and second

section is image processing. There are large number of methods and large number of sources to solve

the above problem. For both shape and colour detection we can use blob detection method and we can

use large number of software for that. Mathematical algorithms are used to analyze the image. Mat lab,

ENVI can use analyze the image. There are so many open source tools. Such as VLFeat,

BoofCV,ILWIS& GRASS. For our project we use OpenCV. OpenCV is an open source computer vision

library. The library is written in C and C++and runs under Linux, Windows and provides interfaces for Python,

Ruby, Matlab and other languages. OpenCV library contains abundant advanced math functions, image

processing functions, and computer vision functions that span many areas in vision.

7

1.2 Problem Description

Our objective is to build a system that can detect, recognize objects according to the colour shape and

barcodes. The whole process should be done in real-time, the thing that necessities employing fast and

efficient algorithms. In addition, the actual implementation of our system should take practicality and

ease of use into account. This is because a system as ours is intended to be used in daily life and hence

needs to be both simple and efficient to use.

1.3 Software and Hardware

1.3.1 Raspberry pi minicomputer

Instead of personal computer we use raspberry pi 2 circuit. Raspberry pi is a series of credit card sized

single board computer. Raspbian-Jessie use as the operating system for compilation and execution of

image processing programs.

Raspberry pi minicomputer is the critical point of our project. It is slower than modern laptop but still

complete Linux computer and can be provided all expected abilities at a low power consumption level.

Raspberry pi is open hardware.

Fig 1. Raspberry pi minicomputer

8

Raspberry pi minicomputer has following specifications

Table 01 –Raspberry pi specifications

Product name Raspberry pi 2 model B

Product description This is the second generation of raspberry pi. It has an upgraded

Broadcom BCM2836 processor, which is powerful ARM cortex-A7

based quad-core processor that runs at 900MHz.The board also features

an increase in memory capacity to 1Gb.

Specifications

chip Broadcom BCM2836 SoC

Core architecture Quad-core ARM Cortex-A7

CPU 900 MHz

GPU Dual core VideoCore Multimedia co-processor provides open GL ES 2.0,

hardware-accelerated openVG AND 1080P30 H.264 high-profile decode

capable of 1Gpixel/s, 1.5Gtexel/s or 24 GFLOPs with texture filtering

and DMA infrastructure.

Memory 1GB LPDDR2

Operating system Boots from Micro SD card, running a version of the Linux operating

system

Dimensions 85 x 56 x 17mm

Power Micro USB socket 5V, 2A

Connectors

Ethernet 10/100 BaseT Ethernet socke

Video capture HDMI ( 1.3 & 1.4)

Audio output 3.5mm jack, HDMI

USB 4 x USB 2.0 Connector

GPIO connector 40-pin 2.54 mm (100 mil) expansion header: 2x20 strip

Providing 27 GPIO pins as well as +3.3 V, +5 V and GND supply lines

Camera connector 15-pin MIPI Camera Serial Interface (CSI-2)

Display connector Display Serial Interface (DSI) 15 way flat flex cable connector

with two data lanes and a clock lane

Memory card slot Micro SDIO

9

1.3.2 Camera System and Illumination Condition

Light became a crucial importance in machine vision. The goal of lighting in machine vision is to

obtain a robust application by:

1. Enhancing the features to be inspected.

2. Assuring high repeatability in image quality.

Illumination is the way an object is lit up and lighting is the actual lamp that generates the

illumination. Light can be ambient, such as normal indoor light or sunlight, or special light that has

been chosen with the particular vision application's needs in mind.

Lighting Types

There is a large variety of different lighting types that are available for machine vision. The types

listed here represent some of the most commonly used techniques. The most accepted type of light for

machine vision is the LED (Light-Emitting Diode), because it’s even light, long life, and low power

consumption.

1. Ring Light

2. Spot Light

3. Backlight

4. Darkfield

5. On-Axis Light

Fig 2. Light room. Fig 3. Ring light. Fig 4. Spot light.

10

We used ring light and spot light method to our project. Spot light has all the light emanating from one

direction that is different from the optical axis. For flat objects, only diffuse reflections should reach

the camera.

A ring light is mounted around the optical axis of the lens, either on the camera or somewhere in

between the camera and the object. The angle of incidence depends on the ring diameter, where the

lighting is mounted, and at what angle the LEDs are aimed.

Different objects reflect different colors. A blue object appears blue because it reflects the color blue.

Therefore, if blue light is used to illuminate a blue object, it will appear bright in a gray scale image. If

a red light is used to illuminate a blue object it will appear dark. It is thus possible to use color to an

advantage, even in gray scale imaging.

Optical Filters

An optical filter is a layer in front of the sensor or lens that absorbs certain wavelengths (colors) or

polarizations. For example, sunglasses have an optical filter to protect your eyes from hazardous UV

radiation. Similarly, we can use a filter in front of the camera to keep the light we want to see and

suppress the rest.

Two main optical filter types are used for machine vision:

1. Band-pass filter: Only transmits light of a certain color, i.e. within a certain wavelength interval. For

example, a red filter only lets red through.

2. Polarization filter: Only transmits light with a certain polarization. Light changes its polarization

when it is reflected, which allows us to filter out unwanted reflections.

Fig 5. Conveyor.

11

1.3.3 OpenCV and Python

OpenCV is an open source computer vision library. The library is written in C and C++and runs under

Linux, Windows and provides interfaces for Python, Ruby, Matlab and other languages. OpenCV

library contains abundant advanced math functions, image processing functions, and computer vision

functions that span many areas in vision

OpenCV library can be divided into several modules

1. The openCV_core module –That contains the core functionalities of the

library,inparticular,basic data structures and arithmetic functions

2. The openCV_imgproc module –That contains the main image processing functions

3. The openCV_highgui module –That contains the image and video reading and writing

functions along with some user interface functions

4. The openCV_feature2d module –That contains the feature point detectors and descriptors and

the feature point matching frame work

5. The openCV_calib3d module –That contains the camera calibratin,two view geometry

estimation and stereo function.

OpenCV provides lot of algorithms related to machine vision system. Python is a general purposes

programming language. It is very simple and readable, compared to other interface languages such as

C/C++ python is lower but very easy to code in python. Numpy is a numerical operations library. All

the openCV array structures are converted to and from numpy arrays. Therefore any operation can be

done in numpy. When considering all above factors openCV-python is an appropriate combinated tool

for fast prototyping of computer vision problems.

12

CHAPTER 2 -Solution Methodology

Considering whole project, we divide this project into six sections as follow

1. Conveyor design and fabrication

2. Install operating systemopenCV and other necessary software packages to raspberry pi

minicomputer

3. Colour detection method

4. Shape detection method

5. Data communication method between mini computer and Database

2.1 Install software to raspberry pi mini computer

Step Description

0 Updating and upgrading installed packages, followed by updating the Raspberry Pi

firmware

1 Install the required developer tools and packages.

2 Install the necessary image I/O packages. These packages allow you to load various

image file formats such as JPEG, PNG, TIFF, etc

3 Install the GTK development library. This library is used to build Graphical User

Interfaces (GUIs) and is required for the highgui library of openCV which allows you

to view images on screen

4 Install the necessary video I/O packages. These packages are used to load video files

using OpenCV

5 Install libraries that are used to optimize various operations within OpenCV

6 Install pip

7 install the Python 2.7 development tools

8 Download OpenCV and unpack it

Setup the build

Compile OpenCV

install OpenCV

13

2.2 Colour Detection Method

There are large number of methods for colour detection in openCV. In our project we use RGB to

HSV colour conversation method for colour detect.

Red, green, blue colour components can be represented as RGB colour format. Colour range is 0 to

255.The RGB colour model is an additive colour model. The main purpose of RGB colour model

sensing, representation and display of images in electronic systems.

HSV is cylindrical colour coordinate. HSV means hue, saturation, value.

Hue is represented the colour wheel.

Fig 6. RGB colour wheel

Saturated is represented the greyness of the colour.so that a Saturation value near 0 means it is dull or

grey looking whereas as a Saturation value of 0.8 might be a very strong color

Fig 7- Relationship between hue and saturation

14

Value is the brightness of the pixel, so 0.1 is black and 0.9 is white.

Fig 8- Relationship among hue saturation value

2.2.1 Steps of Colour Detection

1. Take each frame of the video

2. Convert from BGR to HSV color-space

3. Threshold the HSV image for a range of color

4. Show the mask image.

2.3 Shape Detection Method

There are so many methods to shape detect by using openCV. In our project we use contour method

for that purpose.

Contours can be explained as a curve joining all the continuous points along the boundary. Contour

method is the most useful method for shape analysis. For the better accuracy we use binary images for

that.

2.3.1 Steps for Shape Detection

1. Capture the image

2. Get an image after 30 frames

3. Delete the camera

4. Threshold the image

5. Find contours

6. Approximate contours

7. Show correct shape

15

2.4Process flow chart

16

2.5 Project Layout

Whole project layout can be described as following figure.

Fig 9 –Project layout

2.6 Limitations

There are limitations in our project.

1. Can’t be identify defective items.

2. Limited computer processing speed.

3. There are specified software versions that can compatible for raspberry pi

4. Software limitations.

5. Conveyor speed is limited.

6. Time duration for the project is not enough.

17

CHAPTER 3 -RESULTS

3.1 Colour Detection Results

Fig 10. Red colour identification

Mask 2 identify red colour

Fig 11 Bluecolour identification

Mask identify blue colour

Red Frame

Mask 3 Mask 2

Mask

Blue

Frame

Mask

Mask 2 Mask 3

18

3.2 Shape Detection Results

Fig 12 Square identification

Fig 13Triangular identification

Window

(Square)

Identify

square

Number of

contours=4

Square

Window

(Triangle) Identify

triangular

Number of

contours=3

Triangle

19

Fig 14 Circle identification

Number of

contours=16

Circle

Window

(Circle)

Identify

circle

20

CHAPTER 4 -Deliverables

Build a new system that can detect, recognize objects according to the colour shape. Sole purpose of

outputting their equivalent in string. The whole process should be done in real-time, the thing that

necessities employing fast and efficient algorithms. In addition, the actual implementation of our

system should take practicality and ease of use into account. This is because a system as ours is

intended to be used in daily life and hence needs to be both simple and efficient to use.

21

CHAPTER 5 -Conclusion

In this project, we build a new system that can detect, recognize objects according to the colour shape.

We used opencv and python as image processing software and raspberry pi circuit. We designed a

conveyor for moving objects and we designed new light system to reduce noise. This system can used

for further development in vision based projects.

22

References

HSV colour conversion

Available online http://shervinemami.info/colorConversion.html

Accessed(February 29,2016)

Intro robotics

Available online http://www.intorobotics.com/how-to-detect-and-track-object-with-opencv/

Accessed(February 29,2016)

C. Richard Johnson, Jr., Ella Hendriks, Igor J. Berezhnoy, Eugene Brevdo, Shannon M. Hughes, Ingrid

Daubechies, Jia Li, Eric Postma, and James Z. Wang

Image Processing for Artist Identification, IEEE signal processing magazine july 2008, 1053-

5888/08/$25.00©2008IEEE

Theo Gevers*, Arnold W.M. Smeulders

Color-based object recognition, ISIS, Faculty of WINS, University of Amsterdam, Kruislaan 403,

1098 SJ, Amsterdam, The Netherlands Received 22 December 1997; received for publication 4

February 1998

Y.Ramadevi, T.Sridevi, B.Poornima, B.Kalyani

SEGMENTATION AND OBJECT RECOGNITION USING EDGE DETECTION TECHNIQUES

International Journal of Computer Science & Information Technology (IJCSIT), Vol 2, No 6,

December 2010

23

Appendix

Colour Detection Code

import cv2

import numpyas np

cap= cv2.VideoCapture(0)

while True :

_,frame =cap.read()

hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

lower_blue = np.array([0 , 100, 100])

upper_blue = np.array([ 130,255, 255])

lower_red =np.array([100,150,0])

upper_red =np.array([255,255,150])

lower_green =np.array([100,0,100])

upper_green=np.array([255,150,255])

mask = cv2.inRange(hsv,lower_blue, upper_blue)

mask2= cv2.inRange(hsv,lower_red,upper_red)

mask3= cv2.inRange(hsv,lower_green,upper_green)

output = cv2.bitwise_and(frame,frame,mask = mask)

cv2.imshow('frame',frame)

cv2.imshow('mask',mask)

cv2.imshow('mask2',mask2)

cv2.imshow('mask3',mask3)

k = cv2.waitKey(5) &0xFF

if k== 27:

break

cv2.destroyAllWindows()

cap.release()

24

Shape Detection Code

import numpyas np

import cv2

#import cv2.cv as cv

cap = cv2.VideoCapture(0)

#Capture the image

defget_image():

retval, img = cap.read()

return img

#Get an image after 30 frames

for iin xrange(30):

img = get_image()

print("Takingggg....")

#Save the image

file = "D:/tttt/test.png"

cv2.imwrite(file, img)

#Delete the camera

del(cap)

final_img = cv2.imread("D:/tttt/test.png")

cv2.imshow("final", final_img)

gray = cv2.imread("D:/tttt/test.png",0)

#identifying the object

ret,thresh=cv2.threshold(gray,127,255,1)

img,contours,h=cv2.findContours(thresh,1,2)

for cntin contours:

apporx=cv2.approxPolyDP(cnt,0.01*cv2.arcLength(cnt,True),True)

print len(apporx)

if len(apporx)==3:

print"Triangle"

cv2.drawContours(img,[cnt],0,255,-1)

cv2.imshow('Triangle',img)

eliflen(apporx)==4:

print"Square"

cv2.drawContours(img,[cnt],0,(0,255,0),-1)

cv2.imshow('Square',img)

eliflen(apporx)>15:

print"Circle"

cv2.drawContours(img,[cnt],0,(0,0,255),-1)

cv2.imshow('Circle',img)

cv2.waitKey(0)

cv2.destroyAllWindows()