Chapter 4 Image Segmentation. Outline Introduction to segmentation Point,line and edge detection Thresholding techniques Watershed segmentation.

Chapter 4 Image Segmentation

Outline

Introduction to segmentationPoint,line and edge detectionThresholding techniquesWatershed segmentationRegion based segmentationApplications

2

3

Preview

• Segmentation is to subdivide an image into its component regions or objects.

• Segmentation should stop when the objects of interest in an application have been isolated.

4

Principal approaches

• Segmentation algorithms generally are based on one of 2 basis properties of intensity values– discontinuity : to partition an image based on

sharp changes in intensity (such as edges)– similarity : to partition an image into regions

that are similar according to a set of predefined criteria.

5

Detection of Discontinuities

• detect the three basic types of gray-level discontinuities– points , lines , edges

• the common way is to run a mask through the image

6

Point Detection

• a point has been detected at the location on which the mark is centered if

|R| T

• where – T is a nonnegative threshold – R is the sum of products of the coefficients with

the gray levels contained in the region encompassed by the mark.

7

Point Detection

• Note that the mark is the same as the mask of Laplacian Operation

• The only differences that are considered of interest are those large enough (as determined by T) to be considered isolated points.

|R| T

Example

8

9

Line Detection

• Horizontal mask will result with max response when a line passed through the middle row of the mask with a constant background.

• the similar idea is used with other masks.• note: the preferred direction of each mask is weighted with

a larger coefficient (i.e.,2) than other possible directions.

10

Line Detection

• Apply every masks on the image• let R1, R2, R3, R4 denotes the response of

the horizontal, +45 degree, vertical and -45 degree masks, respectively.

• if, at a certain point in the image

|Ri| > |Rj|,• for all ji, that point is said to be more likely

associated with a line in the direction of mask i.

11

Line Detection

• Alternatively, if we are interested in detecting all lines in an image in the direction defined by a given mask, we simply run the mask through the image and threshold the absolute value of the result.

• The points that are left are the strongest responses, which, for lines one pixel thick, correspond closest to the direction defined by the mask.

12

Example

13

Edge Detection

• we discussed approaches for implementing– first-order derivative (Gradient operator)– second-order derivative (Laplacian operator)

14

Ideal and Ramp Edges

because of optics, sampling, image acquisition imperfection

15

Thick edge• The slope of the ramp is inversely proportional to the

degree of blurring in the edge.• We no longer have a thin (one pixel thick) path.• Instead, an edge point now is any point contained in the

ramp, and an edge would then be a set of such points that are connected.

• The thickness is determined by the length of the ramp.• The length is determined by the slope, which is in turn

determined by the degree of blurring.• Blurred edges tend to be thick and sharp edges tend Blurred edges tend to be thick and sharp edges tend

to be thinto be thin

16

First and Second derivatives

the signs of the derivatives would be reversed for an edge that transitions from light to dark

17

Second derivatives

• produces 2 values for every edge in an image (an undesirable feature)

• an imaginary straight line joining the extreme positive and negative values of the second derivative would cross zero near the midpoint of the edge. (zero-crossing zero-crossing propertyproperty)

18

Noise in Images

• First column: images and gray-level profiles of a ramp edge corrupted by random Gaussian noise of mean 0 and = 0.0, 0.1, 1.0 and 10.0, respectively.

• Second column: first-derivative images and gray-level profiles.

• Third column : second-derivative images and gray-level profiles.

19

Noise in images(contd)

• fairly little noise can have such a significant impact on the two key derivatives used for edge detection in images

• image smoothing should be serious consideration prior to the use of derivatives in applications where noise is likely to be present.

20

Edge point

• to determine a point as an edge point– the transition in grey level associated with the

point has to be significantly stronger than the background at that point.

– use threshold to determine whether a value is “significant” or not.

– the point’s two-dimensional first-order derivative must be greater than a specified threshold.

21

Gradient Operator

• first derivatives are implemented using the magnitude of the gradientmagnitude of the gradient.

y

fx

f

G

G

y

xf

21

22

2122 ][)f(

y

f

x

f

GGmagf yx

the magnitude becomes nonlinearyx GGf

commonly approx.

22

Gradient Masks

23

Diagonal edges with Prewitt and Sobel masks

24

Example

25

Example

26

Example

27

Laplacian

2

2

2

22 ),(),(

y

yxf

x

yxff

(linear operator)

Laplacian operator

)],(4)1,()1,(

),1(),1([2

yxfyxfyxf

yxfyxff

28

Laplacian of Gaussian

• Laplacian combined with smoothing to find edges via zero-crossing.

2

2

2)( r

erh

where r2 = x2+y2, and is the standard deviation

2

2

24

222 )(

r

er

rh

29

Mexican hat

the coefficient must be sum to zero

positive central term surrounded by an adjacent negative region (a function of distance)zero outer region

30

Linear Operation

• second derivation is a linear operation

• thus, 2f is the same as convolving the image with Gaussian smoothing function first and then computing the Laplacian of the result

31

Example

a). Original imageb). Sobel Gradientc). Spatial Gaussian smoothing functiond). Laplacian maske). LoGf). Threshold LoGg). Zero crossing

32

Zero crossing & LoG

• Approximate the zero crossing from LoG image

• to threshold the LoG image by setting all its positive values to white and all negative values to black.

• the zero crossing occur between positive and negative values of the thresholded LoG.

33

Thresholdingimage with dark background and

a light object

image with dark background and two light objects

34

Multilevel thresholding

• a point (x,y) belongs to – to an object class if T1 < f(x,y) T2

– to another object class if f(x,y) > T2

– to background if f(x,y) T1

• T depends on – only f(x,y) : only on gray-level values Global threshold– both f(x,y) and p(x,y) : on gray-level values and its

neighbors Local threshold

35

The Role of Illumination

f(x,y) = i(x,y) r(x,y)

a). computer generated reflectance functionb). histogram of reflectance functionc). computer generated illumination function (poor)d). product of a). and c).e). histogram of product image

easily use global thresholdingobject and background are separated

difficult to segment

36

Basic Global Thresholding

generate binary image

use T midway between the max and min gray levels

37

Basic Global Thresholding• based on visual inspection of histogram1. Select an initial estimate for T.2. Segment the image using T. This will produce two

groups of pixels: G1 consisting of all pixels with gray level values > T and G2 consisting of pixels with gray level values T

3. Compute the average gray level values 1 and 2 for the pixels in regions G1 and G2

4. Compute a new threshold value5. T = 0.5 (1 + 2)6. Repeat steps 2 through 4 until the difference in T in

successive iterations is smaller than a predefined parameter To.

38

Example: Heuristic methodnote: the clear valley of the histogram and the effective of the segmentation between object and background

T0 = 03 iterations with result T = 125

39

Basic Adaptive Thresholding

• subdivide original image into small areas.

• utilize a different threshold to segment each subimages.

• since the threshold used for each pixel depends on the location of the pixel in terms of the subimages, this type of thresholding is adaptive.

40

Example : Adaptive Thresholding

41

Further subdivisiona). Properly and improperly segmented subimages from previous example b)-c). corresponding histogramsd). further subdivision of the improperly segmented subimage.e). histogram of small subimage at topf). result of adaptively segmenting d).

Segmentation by Morphological Watersheds

Introduction• Based on visualizing an image in 3D

020

4060

80100

0

50

1000

5

10

15

20

25

imshow(I,[ ]) mesh(I)

• Instead of working on an image itself, this technique is often applied on its gradient image.– In this case, each object is

distinguished from the background by its up-lifted edges

020

4060

80100

0

20

40

60

80

1000

2

4

6

8

Basic Definitions

• I: 2D gray level image• DI: Domain of I• Path P of length l between p and q in I

– A (l +1)-tuple of pixels (p0=p,p1,…,pl=q) such that pi,pi+1 are adjacent (4 adjacent, 8 adjacent, or m adjacent, see Section 2.5)

• l(P): The length of a given path P• Minimum

– A minimum M of I is a connected plateau of pixels from which it is impossible to reach a point of lower altitude without having to climb

Basic Definitions• Instead of working on an image itself, this technique is often applied

on its gradient image.• Three types of points

– Points belonging to a regional minimum– Catchment basin / watershed of a regional minimum

• Points at which a drop of water will certainly fall to a single minimum– Divide lines / Watershed lines

• Points at which a drop of water will be equally likely to fall to more than one minimum

• Crest lines on the topographic surface• This technique is to identify all the third type of points for

segmentation

Basic Steps1. Piercing holes in

each regional minimum of I

2. The 3D topography is flooded from below gradually

3. When the rising water in distinct catchment basins is about to merge, a dam is built to prevent the merging

The dam boundaries correspond to the watershed lines to be extracted by a watershed segmentation algorithm

- Eventually only constructed dams can be seen from above

Dam Construction

• Based on binary morphological dilation• At each step of the algorithm, the binary

image in obtained in the following manner1. Initially, the set of pixels with minimum gray

level are 1, others 0.2. In each subsequent step, we flood the 3D

topography from below and the pixels covered by the rising water are 1s and others 0s. (See previous slides)

Notations• M1, M2:

– Sets of coordinates of points in the two regional minima

• Cn-1(M1), Cn-1(M2)– Sets of coordinates of points

in the catchment basins associated with M1 M2 at stage n-1 of flooding (catchment basins up to the flooding level)

• C[n-1]– Union of Cn-1(M1), Cn-1(M2)

Dam Construction• At flooding step n-1, there are

two connected components. At flooding step n, there is only one connected component

– This indicates that the water between the two catchment basins has merged at flooding step n

– Use “q” to denote the single connected component

• Steps– Repeatedly dilate Cn-1(M1), Cn-1(M2)

by the 3×3 structuring element shown, subject to the following condition

• Constrained to q (center of the structuring element can not go beyond q during dilation

Dam Construction• The dam is constructed by

the points on which the dilation would cause the sets being dilated to merge.

– Resulting one-pixel thick connected path

• Setting the gray level at each point in the resultant path to a value greater than the maximum gray value of the image. Usually max+1

Watershed Transform• Denote M1, M2, …, MR as the sets of the coordinates of the points in

the regional minima of an (gradient) image g(x,y)

• Denote C(Mi) as the coordinates of the points in the catchment basin associated with regional minimum Mi.

• Denote the minimum and maximum gray levels of g(x,y) as min and max

• Denote T[n] as the set of coordinates (s,t) for which g(s,t) < n

• Flood the topography in integer flood increments from n=min+1 to n=max+1

• At each flooding, the topography is viewed as a binary image

Watershed Transform• Denote Cn(Mi) as the set of

coordinates of points in the catchment basin associated with minimum Mi at flooding stage n.– Cn(Mi)= C(Mi) T[n]

– Cn(Mi)=T[n]

• Denote C[n] as the union of the flooded catchment basin portions at stage n:

• Initialization– Let C[min+1]=T[min+1]

• At each step n, assume C[n-1] has been constructed. The goal is to obtain C[n] from C[n-1]

C(n)R

ii

R

iin MCCMCnC

11

)(]1[max and )(][

Watershed Transform• Denote Q[n] as the set of connected components in T[n]. • For each qQ[n], there are three possibilities

1. q C[n-1] is empty (q1)

• A new minimum is encountered• q is incorporated into C[n-1] to form C[n]

2. q C[n-1] contains one connected component of C[n-1] (q2)

• q is incorporated into C[n-1] to form C[n]

3. q C[n-1] contains more than one connected components of C[n-1] (q3)

• A ridge separating two or more catchment basins has been encountered

• A dam has to be built within q to prevent overflow between the catchment basins

4. Repeat the procedure until n=max+1

Examplea: Original imageb: Gradient image of image ac: Watershed lines obtained

from image b (oversegmentation)

Each connected region contains one local minimum in the corresponding gradient image

d: Watershed lines obtained from smoothed image b

The Use of Markers• Internal markers are used to limit the number of regions by

specifying the objects of interest

– Like seeds in region growing method

– Can be assigned manually or automatically– Regions without markers are allowed to be merged (no

dam is to be built)

• External markers those pixels we are confident to belong to the background

– Watershed lines are typical external markers and they belong the same (background) region

Watershed Based Image Segmentation

1. Use internal markers to obtain watershed lines of the gradient of the image to be segmented.

2. Use the obtained watershed lines as external markers3. Each region defined by the external markers contains a single

internal marker and part of the background4. The problem is reduced to partitioning each region into two

parts: object (containing internal markers) and a single background (containing external markers)

– Global thresholding, region growing, region splitting and merging, or watershed transform

59

Region-Based Segmentation - Region Growing

• start with a set of “seed” points

• growing by appending to each seed those neighbors that have similar properties such as specific ranges of gray level

60

Region Growing

select all seed points with gray level 255

criteria:1. the absolute gray-

level difference between any pixel and the seed has to be less than 65

2. the pixel has to be 8-connected to at least one pixel in that region (if more, the regions are merged)

• Region growing: Groups pixels or sub-region into larger regions.– step1:

• Start with a set of “seed” points and from these grow regions by appending to each seed those neighboring pixels that have properties similar to the seed.

– step2:• Region splitting and merging

61

Region-based segmentation

• Advantage:– With good connectivity

• Disadvantage:– Initial seed-points:

• different sets of initial seed-point cause different segmented result

– Time-consuming problem

62

Region-based segmentation

• Unseeded region growing:– no explicit seed selection is necessary, the

seeds can be generated by the segmentation procedure automatically.

– It is similar to SRG except the choice of seed point

63

Region-based segmentationUnseeded region growing

• Advantage:– easy to use– can readily incorporate high level knowledge of

the image composition through region threshold

• Disadvantage:– slow speed

64

Region-based segmentationUSRG

Region-based segmentationfast scanning

• Fast scanning Algorithm: – The fast scanning

algorithm somewhat resembles unseeded region growing

– the number of clusters of both two algorithm would not be decided before image passing through them.

65

66

Region-based segmentationfast scanning

Region-based segmentationfast scanning

• Last step:

– merge small region to big region

67

• Advantage:– The speed is very fast– The result of segmentation will be intact with good

connectivity

• Disadvantage:– The matching of physical object is not good

• It can be improved by morphology and geometric mathematic

68

Region-based segmentationfast scanning