Computer vision techniques for interactive art

Computer Vision Techniques forInteractive Art

Using Processing and Max/Jitter

Interactive Art Doctoral Programhttp://www.artes.ucp.pt/si/doutoramento/

index.html

10-04-2023 Jorge Cardoso 1

Interactive Art Doctoral Program

Topics• Techniques

– Frame differencing (for motion detection)– Color detection (where in the image, does a given

color appear – can be used for tracking objects)– Brightness detection (what is the position of the

brightest pixels)– Background subtraction (for object segmentation) – Blob extraction (applied to the previous techniques to

extract shape parameters and vectorizing silhuettes)– Movement estimation (what direction are the objects

moving)– Face detection (is there a face in the image? where?)

10-04-2023

Jorge Cardoso

2


How• General overview of how those CV

techniques work • Examples of what can be accomplished

with the techniques– Vídeos– Live demos

• Code examples of these techniques in Processing and Jitter• Some – very simple(istic) – examples of how they can be

used for interactive work

10-04-2023

Jorge Cardoso

3


Computer Vision for Interaction

• Use a computer and camera to extract information about what people are doing and use that as an implicit or explicit form of interaction

• “Computer vision is the science and technology of machines that see. As a scientific discipline, computer vision is concerned with the theory for building artificial systems that obtain information from images” – Computer vision. (2009, April 24). In Wikipedia, The Free Encyclopedia. Retrieved 11:53, May 9, 2009, from http://en.wikipedia.org/w/index.php?title=Computer_vision&oldid=285848177

10-04-2023

Jorge Cardoso

4


10-04-2023

Jorge Cardoso

5

Image• Computer vision are processes applied to

sequences of images– An image is just a sequence of number (usually

arranged in a matrix form)• Each number represents a colour

50 44 23 31 38 52 75 5229 09 15 08 38 98 53 5208 07 12 15 24 30 51 5210 31 14 38 32 36 53 6714 33 38 45 53 70 69 4036 44 58 63 47 53 35 2668 76 74 76 55 47 38 3569 68 63 74 50 42 35 32


Image

• Each pixel (i.e., number in the matrix) is usually stored in RGB or ARGB format so– It can be decomposed:

10-04-2023

Jorge Cardoso

6

or


Frame Differencing

• If we subtract two consecutive video frames, threshold and binarize it, we get a rough view of the movement in the video

• Procedure:– Go through the current frame image, pixel by pixel– Calculate the distance between the color of the current frame’s

pixel and the previous frame’s pixel– If distance is smaller than a given threshold, assume no

movement (black), otherwise assume movement (white)

10-04-2023

Jorge Cardoso

7


Frame Differencing• Color difference

– Q: How do we calculate the “distance” between two colors?

– A: Simple version: euclidean distance.• Take a color as a point in 3D space (RGB -> XYZ)• Calculate the distance between the two points

– Subtract each component (dR = R1-R2, dG = G1-G2, dB = B1-B2)

– Dist = Sqrt(dR*dR + dG*dG + dB*dB)

– A1: Slightly (perceptively) better version: weight each component differently:• Dist = Sqrt(2*dR*dR + 4*dG*dG + 3*dB*dB)

10-04-2023

Jorge Cardoso

8


Frame Differencing• Q: What’s it good for?• A:

– KALOETHOGRAPHIA (2010) • Pedro Cascalheira • http://artes.ucp.pt/blogs/index.php/vai/2011/02/14/alunos-

2009-2010

– aB (2011) • Jorge Coutinho• https

://picasaweb.google.com/lh/photo/x9bGyxFsTLVFbkk9kD3xLA?feat=directlink

– WebCam Piano • Memo Akten• [video]

10-04-2023

Jorge Cardoso

9

https://picasaweb.google.com/lh/photo/x9bGyxFsTLVFbkk9kD3xLA?feat=directlink




Frame Differencing - Code

• Processing Code– FrameDifferencing– FrameDifferencingWithMotionDetection– FrameDifferencingForAirPiano

• Jitter Code– FrameDifferencing.maxpat– FrameDifferencingWithMotionDetection.

maxpat

10-04-2023

Jorge Cardoso

10


Color detection

• Isolate color regions in an image• Procedure:

– Go through the image, pixel by pixel– Calculate the distance between the color of the

current pixel and the reference color– If distance is smaller than a given threshold,

keep the pixel10-04-2023

Jorge Cardoso

11


Color Detection• Color Difference

– For a better approach than RGB distance, use HSB

– Give more weight to the Hue– Dist = Sqrt(3*dH*dH + dS*dS +

2*dB*dB)

10-04-2023

Jorge Cardoso

12


Color Detection

• Q: What’s it good for?• A:

– Catch of the day [video]– PlayDohAsInterface Keyboard [video]

10-04-2023

Jorge Cardoso

13


Color Detection - Code

• Processing Code– ColorDifference– ColorDifferenceSequencer

• Jitter Code– ColorDifferenceWithFindbounds.maxpat– ColorDifference.maxpat

10-04-2023

Jorge Cardoso

14


Brighteness detection

• Slight variation on color detection when we’re only interested in tracking bright pixels– For example, to track a light

• Procedure:– Same as color detection but extract the brightness of

each pixel and keep only the highest

10-04-2023

Jorge Cardoso

15


10-04-2023

Jorge Cardoso

16

Brighteness detection• Q: How do we calculate the “brightness” of

a color?• A:

– In RGB:• (perceptive) Brightness = (0.2126*R) + (0.7152*G) +

(0.0722*B) • (physical/energy) Brightness = (R+G+B)/3• …

– In HSB:• Brightness = B

– Just use the brightness() function in whatever language…


Brightness detection

• Q: Why not just color detection?• A:

– In some situations you don’t care about color, just the brightness (for example to detect a lantern)

– Brightness detection is more robust (i.e., less influenced by lighting conditions)

10-04-2023

Jorge Cardoso

17


10-04-2023

Jorge Cardoso

18

Brighteness detection


–Drawing With Light [video]–Burning the Sound [video]


Brighteness detection - Code• Processing Code

– BrightnessDetection• Jitter Code

– BrightnessDetection.maxpat

10-04-2023

Jorge Cardoso

19


Background subtraction

10-04-2023

Jorge Cardoso

20

• If we have a fixed camera, and a background image for reference, we can subtract the background from each frame to isolate new objects:

This - This = This



• Q: How do we “subtract” two images?

• A: Pixel by pixel

• Q: How do we “subtract” two pixels?• A: Channel by channel (color

component by color component, usually R, G and B)

10-04-2023

Jorge Cardoso

21



• Pixel subtraction alone doesn’t work very well due to noise or shadows in the image

• The resulting subtraction must be thresholded to eliminate noise

10-04-2023

Jorge Cardoso

22



• After subtracting and thresholding, we can binarize the image to get a mask

10-04-2023

Jorge Cardoso

23


Background subtraction• Q: What’s it good for?• A: Isolating objects in a scene when you

don’t have much control over the background. – Background Subtraction Processing Demo

[video]

10-04-2023

Jorge Cardoso

24


Background subtraction - Code

• Processing code:– BackgroundSubtraction

• Jitter code– BackgroundSubraction.maxpat

10-04-2023

Jorge Cardoso

25


Blob extraction

• All the previous techniques result in possibly many white areas distributed in the image

10-04-2023

Jorge Cardoso

26

http://en.wikipedia.org/wiki/Connected_Component_Labeling


Blob extraction

• Blob extraction (blob detection, connected component labeling) identifies those regions and and allows the extractions of some features – area, – perimeter, – orientation, – bounding box, – Silhouette

10-04-2023

Jorge Cardoso

27


Blob extraction

• Procedure:– See http://en.wikipedia.org/wiki/

Connected_Component_Labeling

10-04-2023

Jorge Cardoso

28




Blob extraction


– Loliness (2007) • Telmo Rocha [video]

– Células (2009) • Ana Lima [video]

– SilhouetteCollisions [video]

10-04-2023

Jorge Cardoso

29


Blob extraction - Code

10-04-2023

Jorge Cardoso

30

• Processing libraries


Blob extraction - Code• Processing Code (with Flob library)

– BlobExtractionFromFrameDifferencing– BlobExtractionAfterColorDifference

• Processing Code (with Jmyron library)– Em Mac Intel é preciso substituir o ficheiro libjmyron por este:

http://www.jibberia.com/projects/

– BlobExtractionAfterBackgroundSubtraction

10-04-2023

Jorge Cardoso

31


Blob extraction - Code

• Jitter Code (com external cv.jit)– BlobExtractionAfterBrightnessDetection

10-04-2023

Jorge Cardoso

32


Motion estimation (aka Optical Flow)

10-04-2023

Jorge Cardoso

33

• Estimate the movement vectors of blocks in a sequence of two images

• Procedure (naive)– Divide both images

into blocks– For each block in

image one, find the closest (more similar) block in image two

– http://en.wikipedia.org/wiki/Lucas%E2%80%93Kanade_method

Image: http://grauonline.de/wordpress/wp-content/uploads/ovscreenshot1.png

http://grauonline.de/wordpress/wp-content/uploads/ovscreenshot1.png


Motion estimation

10-04-2023

Jorge Cardoso

34


– opticalflow-particles [video]– opticalflow-navigation [video]


Motion estimation - Code• Jitter Code

– OpticalFlow.maxpat

10-04-2023

Jorge Cardoso

35


Face detection

• Not recognition!• Needs a configuration file that determines

what to detect (front faces, side faces, body, etc)

• Returns the position and size of detected faces

10-04-2023

Jorge Cardoso

36


Face Detection


– Anatomias Urbanas (2008)• Sara Henriques [video]

– Ex.001 (2010)• Bernardo Santos • [foto] https://picasaweb.google.com/lh/photo/_KCdCQGUeEw8lSzF8ZMPVw?

feat=directlink

10-04-2023

Jorge Cardoso

37


Face Detection - Code

• Processing (with OpenCV processing library)– FaceDetection– FaceDetectionFunnyEyes

• Jitter (with cv.jit)– FaceDetectionWithASmiley.maxpat

10-04-2023

Jorge Cardoso

38


Computer Vision Techniques for Interactive Art

• Not a recent area• Myron Krueger experimented with some of

these techniques in the 70s [video Myron Krueger]

10-04-2023

Jorge Cardoso

39


The End

10-04-2023

Jorge Cardoso

40

http://slideshare.net/jorgecardoso (tag: ea-phd-ia-2011)

Computer vision techniques for interactive art

Education

Computer vision techniques for interactive art