Robotics: Integrated Systems Design

Robotics: Integrated Systems DesignToday: Vision

Reading: Murphy, Section 6.6Wednesday: Team Work Day

How does the human eye work?

Color Perception in Human

• Two kinds of light receptors: rods and cones• Rods register the relative light and dark• Cones perceive color

– Three kinds of cones: red, green, blue

Computer Vision• Uses the electromagnetic spectrum to

produce an image.– Visible light, x-rays, thermal, infrared

Representation• Image: picture like format where there is a direct physical

correspondence to the scene being viewed.– There are multiple readings in a grid

• Pixels: “picture element”– Measure depends on the type of spectra being used– Image function: converts a signal to a pixel value

CCD Cameras• Charged-Couple Device: detects visible

light– Light falls on an array of metal-oxide

semiconductor capacitor (MOS)– Line transfer or frame transfer– Frame grabber

• A/D conversion

– A/D conversion • Slower than CCD • Frame buffers

Representations

• Grayscale– 8-bit– 256 discrete gray values– 0 black, 255 white

Representations

• Different method for representing color– RGB space: red, green, blue– HSV space: hue, saturation, and value

RGB

• Color Space– 24 bit color (8 bits

per color)

RGB Representations

• Interleaved– RGB values stored together

• Red = image[row][col][0]• Green = image[row][col][1]• Blue = image[row][col][2]

• Separate– RGB values stored as separate 2D arrays

• Red = image_red[row][col]• Green = image_green[row][col]• Blue = image_Blue[row][col]

Region Segmentation

Region Segmentation• Identifying a region in an image with a particular color

– Thresholding– Binary image

HSV Representations

– HSV space: hue, saturation, and value• Hue is the dominant wavelength• Saturation is the lack of whiteness• Value is intensity or brightness the light

– Linear transformation between RGB and HSV

HSV Color Space

Comparison of Region Segmentation

Original Image

RGB segmentation

HSV Segmentation

IC Color Vision System

• Color Space

• Finding color blobs in an image

• Making a color model

• Tuning the camera

Color Selection Plane

Hue=0

Hue=360

Sat=0Val=224

Sat=224Val=224

Sat=224Val=0

Note: 224 is the range of values the camera pixels put out in each of R, G & B

Color Blobs• For color tracking, a rectangular piece of the color

selection plane is selected. All of the pixels in the image whose color falls within that piece are selected.

• Selected pixels that are contiguous are combined as blobs

• Each blob has a size, position, number of pixels, major and minor axis, etc.

• The blobs correspond to objects seen in the image that are the desired color (as given by the specified piece of the of the color selection plane).

Color Models• The XBC can segment the image using three

different pieces of the color selection plane (each is called a color model) simultaneously

• It can track a number of blobs from each color model• It can display the video in any one of three ways

– Raw (live video)– Processed (pixels not selected by a color model are dark)– Combined (processed combined with raw show you can

see where the blobs are on the live video)

More on Color Models• A Color Model-HSV specifies a bounding box in the color

selection plane • Moving either edge towards the center line constrains the

range of accepted color values to only include more vivid colors.

• If everything you want is being accepted but so is a lot of other junk you don't want, move the corners closer to the center.

Onscreen Instructionsfor setting color models

Onscreen Instructionsfor setting color models

Move ModeMode Designator

Press the SELECT button to view the onscreen instructions

Corner move mode is selected by the L or R button and is indicated by a highlighted corner

Symmetric Move or Resize mode (M or R) is selected by the START button

Processed Video

The A button cycles through live, processed, and combined video

XBC Camera Check1. Create a color model for channel 0 that sees

a red screw driver2. Load xbctest.ic onto your XBC3. Run the program4. Select the vision test5. Select the correct channel/model to see red6. Follow onscreen directions to get data on the

blobs7. If you like your model, save it to flash8. For more info, see XBC Camera in IC Help

Which Color Model?

• The XBC stores 3 channels of color (0, 1, 2)• There are 3 models maintained for each

channel:1. The currently active model (the one you see in

the modify model menu)2. The model in flash (you can save the currently

active model and camera settings using the save to flash menu item or make them active by loading from flash)

3. The default model and camera settings which are made active by selecting restore to default

Tuning the Camera

• By default cameras turn on Auto White Balance (AWB) and dynamically adjust their color temperature to keep the amounts of red and blue in the field of view roughly balanced.  This is fine if the content of the field of view is roughly balanced between red and blue and if the goal is to look good to humans in changing, arbitrary lighting situations.  However, when trying to do color tracking, dynamically changing color temperature is generally counter productive. 

• Instead, you should use the Vision/Camera Config menu and press the Start button to interactively calibrate the white balance while pointing the camera at a white sheet of paper.  This will adjust the Red and Blue components of the color temperature until the amount of each in the scene balances, then turn off AWB to lock those values in.  After this procedure, it's a good idea to go to the Vision/Flash Memory menu, select "Setting: < Camera Config >", and then "Save to Flash".

Tuning the Camera

• Focus the un-focus the lens – Depending on what you are looking for it can

help to have the image a little out of focus

Other Tips

• Blobs are ordered based on size

• track_update– Becareful where this is called

• track_set_minarea– Reduce the processing time

• track_count– Are there any blobs of a particular color

model?

Feature Extraction

• Line detection

• Face detection

• Stereo vision– Depth and range