Python in Computer Vision

Motivation & BackgroundComputer Vision in Python

More InformationSummary

Introduction to using Python in Computer VisionKiwi PyCon, Christchurch, 2009

Brian Thorne

University of Canterbury

6th November 2009

Brian Thorne Computer Vision in Python



Outline

1 Motivation & BackgroundWhat is Computer Vision?Uses & Examples

2 Computer Vision in PythonToolsImage AcquisitionImage FilteringBackground SubtractionFeature Point Detection

3 More InformationDifferent platformsAdditional Tools




Computer VisionUses & Examples

Outline








Vision

25% of the whole brain is for vision. Around 50% of cerebralcortex is for vision, 80% of the brain is associated with visionin some manner.


Computer Vision

DefinitionThe goal of computer vision is to recognize objects and their motion

What is it used for?Scene reconstructionEvent detectionVideo trackingObject recognitionLearningIndexingMotion estimationImage restoration

Computer Vision

DefinitionThe goal of computer vision is to recognize objects and their motion

What is it used for?Scene reconstructionEvent detectionVideo trackingObject recognitionLearningIndexingMotion estimationImage restoration




Computer Vision crosses over with many domains





What makes it hard?

What we see What the computer sees





Vision is inferential

http://web.mit.edu/persci/people/adelson/checkershadow_illusion.html





Outline








Visual Object Classes Challenge 09

http://www.pascal-network.org/challenges/VOC/voc2009





Object Recognition and Segmentation - Texture

−−−−−−−→

(Sharon, Balun, Brandt, Basri)





Object Recognition and Segmentation - Edges

http://www.robots.ox.ac.uk/~vdg/dynamics.html





Traffic Monitoring





Augented Reality - Sixth Sense

’SixthSense’ is a wearable gestural interface that augments thephysical world around us with digital information and lets us use

natural hand gestures to interact with that information.

http://www.pranavmistry.com/projects/sixthsense/




ToolsImage AcquisitionImage FilteringBackground SubtractionFeature Point Detection

Outline





Python In Computer Vision: OpenCV

Provides well tested, optimized andopen source code for image processingand computer visionWritten in C, ensuring both fast andportable.Has been compiled for manyembedded platformsHas multiple language wrappersincluding 3 for PythonTools have been made to use graphicshardware to accelerate CVperformance on the GPU

Project home page and documentation is at:http://opencv.willowgarage.com
















Python In Computer Vision: Numpy & Scipy

Gives strongly typed N-dimensionalarrays to PythonWell used and tested libraries forscientific computingIncludes lots of handy tools such asoptimisation and signal processingused often in computer vision.Usually used with iPython andmatplotlib

SciPy can be downloaded from: http://www.scipy.org

































Python In Computer Vision: Pygame

Game development frameworkNow has basic Computer Vision supportBeing Python it can be used with other Python tools -integrates well with numpy/scipy

pygame can be downloaded from: http://www.pygame.org



















Pycam

This is the project with all the examples for this presentation. Hasa bunch of simple examples like filtering and backgroundsubtraction, face detection.

Contains two video player classes that can work with differentbackend setups, and can incorporate optional processfunctions.Examples of intergrating OpenCV with pygame - eg for eyeand face detection.OpenCV camera class that allows an opencv camera to beused with pygame (No longer required in latest pygame)

VideoCapturePlayerFor the rest of this presentation, examples will use the videocapture code (with error checking) from pycam.

http://pycam.googlecode.comBrian Thorne Computer Vision in Python




Pycam








Pycam








Pycam








Outline








Acquiring & Display Of An Image

Live image acquisition is such a crucialrole in the majority of CV applications.

Example getting and showing a frameas a most basic, but necessary test





Acquiring & Display Of An Image

Live image acquisition is such a crucialrole in the majority of CV applications.

Example getting and showing a frameas a most basic, but necessary test





Python OpenCV: Image Capture

Examplefrom opencv import highgui as hgcapture = hg.cvCreateCameraCapture(0)hg.cvNamedWindow("Snapshot")frame = hg.cvQueryFrame(capture)hg.cvShowImage("Snapshot", frame)hg.cvWaitKey(10000)





Outline








Filtering - Gaussian Blur

One of the simplest operations in image processing is blurringan image

Reduce noise,Remove artifacts

Scale an image“cleanly”

Create motion blur -if done in onedirection

OpenCV includes a gaussian filter among many others(cvSmooth function)SciPy has a multi-dimensional Gaussian filter that acts on aNumPy arrayOr you could convolve an image with a filter manually























































Applying a Gaussian Blur with OpenCV

OpenCV Gaussian Blurfrom pycam import VideoCapturePlayer as VCPfrom opencv import cv

def gaussianBlur(im, filterSize=43):result = cv.cvCreateMat(im.rows, im.cols,

im.type )cv.cvSmooth(image,result,

cv.CV_GAUSSIAN, filterSize)return result

if __name__ == "__main__":VCP(gaussianBlur, "Guassian Filter").main()





Applying a Gaussian Blur with Scipy

SciPy Gaussian Blur

from scipy.ndimage.filters import gaussian_filterfrom pycam import OpencvVideoCapturePlayer as VCPfrom misc import scipyFromOpenCV

@scipyFromOpenCVdef gaussianBlur(np_image):

result = gaussian_filter(np_image,sigma=(4, 4, 0),order=0, mode=’reflect’)

return result

if __name__ == "__main__":VCP(gaussianBlur,"Scipy Guassian Blur").main()





Outline








Background Subtraction

In video security the camera mostly looks at the same boringbackgroundWhat we are usually interested in is when objects (eg people orvehicles) enter or exit a sceneAim is to isolate the interesting, and ignore the boringAt the most simple level background subtraction is simply acomparison between two image framesAt the more complex level many people have gotten phd’s forbetter background learning techniques, and better differencingalgorithms


















Simple Frame Differencing1 To let the camera adjust to light levels, ignore the first few

frames.2 Store a frame as the base frame.3 For each new frame that comes in:

1 Take the absolute intensity difference in each channel(Red/Green/Blue) with the base frame.

2 Binary threshold to ignore pixels that are only a bit different.3 Convert difference image to a one channel mask4 Clean up small noise areas in the mask (with median filter,

erode, connected components)5 Return the changed pixels from the original image using the

created mask.











created mask.











created mask.











created mask.











created mask.











created mask.











created mask.











created mask.





Background Subtract

Here I have placed a cellphone on mycluttered deskCan’t tell thats there is no green screenQuick demo





Outline








Feature Point Detection

Feature point detection is implementedin OpenCV you can do it in one call:cvCornerHarris or cvGoodFeaturesTo demonstrate the algorithm though -we will go look at it in scipy.Implementation derived from JanSolem


Feature Detection

1 First convert to a grey scaleimage

2 Showing the derivative in the xand y directions

3 showing the millions of points ofinterest

4 filtering them




Augmented Reality

Augmented reality is undergoing massivegrowthOpenCV provides the face detectionAn AR game can easily be made in Pygameusing the webcam and face location as theinterface




Different platformsAdditional Tools

Outline








Running on an OLPC

OLPC - provide children in developingnations with access to knowledge, andopportunities to "explore, experimentand express themselves"Includes Python and a webcam - thatsall you need for computer vision!Here I am running OpenCV’sfacedetection on the XO laptopLots of Computer Vision work on theXO has been done using pygame byNirav Patel (http://eclecti.cc/olpc)





Running on an OLPC






Running on an OLPC






Running on an OLPC






Outline





IPython & MatPlotLib

Using IPython, an interactive shell can be used from deepinside a nested loop in a running program.

In the code addfrom IPython.Shell import IPShellEmbed...IPShellEmbed()()

Example

In [1]: from opencv import cvIn [2]: cv.cvAnd(diffImage,image, temp)In [3]: timeit cv.cvAnd(diffImage,image, temp)1000 loops, best of 3: 229 µs per loopIn [4]: from pylab import imshow, showIn [5]: imshow(temp)Out[5]: <AxesImage object at 0x42489d0>In [6]: show()




Documentation & Support

The documentation in both SciPy and OpenCV was found to bepretty good.... not entirely complete. The OpenCV book is reallygood.

Remember Python is FreeDocumentation is not going to be as extensive as for a professionalpackage like Matlab.... but you can help!

Support for these open source packages is almost entirely reliant onexperienced members of the community responding to requests onmessage boards or mailing lists.




Summary

For the scholar of computer vision research, using Python canhelp in trying out new algorithms very quickly. The breadth ofthe additional libraries available and the ease of integrating,make new and novel solutions quickly realizable.For someone just wanting to play around with some cool stuff,its easy to dive in!Limitations on using Python for CV

A major limitation of using Python would be when theapplication is being developed for special embedded hardwareor when the best possible performance is required (at YOURexpense)


References

Thank You!

Thank you toRaphaël Grasset - supervisor at HitLabNZRichard Green - computer vision lecturerJohn Graves & Cristiano Soares for giving me detailed andhelpful feedback


References

For Further Reading I

Library URLPygame http://pygame.orgOpenCV http://opencv.willowgarage.com

Numpy/Scipy http://scipy.orgPycam http://pycam.googlecode.org

G. Bradski, A. KaehlerLearning OpenCV.O’Reilly Media, September 2008.

T. OliphantGuide to NumPy.UT, Trelgol Publishing, 2006.


Python in Computer Vision

Technology

goal of computer vision

brian thorne computer

computer vision kiwi

computer visiondenition

htmlbr thornecomputer

image processing

digital information

examplessummary vision25