Feature Point Detection and Curve Approximation for Early Processing of Free-Hand Sketches Tevfik Metin Sezgin and Randall Davis MIT AI Laboratory.

Feature Point Detection and Curve Approximation for Early

Processing of Free-Hand Sketches

Tevfik Metin Sezgin and Randall Davis

MIT AI Laboratory

Metin Sezgin

Outline

• Motivation and problem statement• Feature point detection

- Curvature and speed as information sources- Average based filtering; Scale space filtering- Hybrid fit generation- Handling strokes with curves

• Evaluation• Related work• Future work

Metin Sezgin

Terminology

•Sketch: Informal,messy diagramsconsisting of several strokes

•Stroke: Array of timestampedpoints

•Feature Point: Cornersof a stroke (vertices)

Metin Sezgin

Task• Given a freehand stroke, generate a

description of the stroke in terms of geometric primitives such as circles, lines, and curves

Metin Sezgin

Lifecycle of a stroke

Data Collection

Data Collection

Vertex DetectionVertex

Detection

PolylineFit

PolylineFit

Oval FitOval Fit

Curve Detection

Curve Detection

Line Fit

Line Fit

Complex Fit(lines and

curves)

Complex Fit(lines and

curves)Best Fit

SelectionBest Fit

Selection OutputOutput

Metin Sezgin

Lifecycle of a stroke

Data Collection

Data Collection

Vertex DetectionVertex

Detection

PolylineFit

PolylineFit

Oval FitOval Fit

Curve Detection

Curve Detection

Line Fit

Line Fit

Complex Fit(lines and

curves)

Complex Fit(lines and

curves)Best Fit

SelectionBest Fit

Selection OutputOutput

Metin Sezgin

Collecting Strokes

Metin Sezgin

Vertex detection

Metin Sezgin

Vertex detection

Curvature

Metin Sezgin

Vertex detection

Curvature Speed

Metin Sezgin

Problems with local extrema

• Input is noisy- Digitization noise- Hand jittering (tremor)

• Picking all local extrema introduces many false positives

• Picking the global extremum gives us a single point

• Picking the n-best extrema doesn’t help either

Metin Sezgin

Solutions

• Average-based filtering- Compute threshold based on data average- Partition using this threshold- Select global extremum in each partition

Metin Sezgin

Average-based filtering

Curvature Speed

Metin Sezgin

Average-based filtering

• Pros- Reliable (when data isn’t too noisy)- Fast: O(n)- False positives easy to filter out

• Cons- Still false positives- Need pre-set constant for scaling

Metin Sezgin

Scale-space filtering

• Real world data has features at many scales

• Represent the data multiple different scales

• Selecting scale(s) at which data is meaningful

Metin Sezgin

Deriving the scale space

Convolve the original signal with Gaussian signals of increasing width

Feature points in the curvature scale space

Metin Sezgin

Selecting a scale

Metin Sezgin

Selecting a scale

Metin Sezgin

Scale selected by the algorithm

The original strokeOutput generated at the

scale chosen by our algorithm

Metin Sezgin

Handling extremely noisy strokes (cont.)

Curvature fit using SSF, 9 fps Speed fit using SSF, 7 fps

Metin Sezgin

Combining information sources

• Individual detection methods may miss points

Speed fit missing a point

Curvature fit missing a point

• Need a way to combine information provided by each method

• Generate a series of hybrid fits and pick the best

Metin Sezgin

Hybrid fit generation

• Given N fits generated using N different methods- Within each fit, compute vertex certainties and sort by

certainty

• Initial fit contains points common to all methods• Generate N new fits by adding best candidate

from each method. • Select the one of these with the least error• Terminate when error smaller than preset error

bound

Metin Sezgin

Data Collection

Data Collection

Vertex DetectionVertex

Detection

PolylineFit

PolylineFit

Oval FitOval Fit

Curve Detection

Curve Detection

Line Fit

Line Fit

Complex Fit(lines and

curves)

Complex Fit(lines and

curves) Best FitSelectionBest Fit

Selection OutputOutput

Metin Sezgin

Curve approximation

c2

c1

uv

Metin Sezgin

Results

Metin Sezgin

Results

Metin Sezgin

User study results

• All users except one perceived the system as more natural.

• Overall it took users much less time to specify geometries.

• All subjects except one indicated they would prefer an interface that feels like our system for specifying shapes in a sketchy fashion.

• Strokes classified correctly 96% of the time

Metin Sezgin

Related work

• Existing systems lack- Support for arbitrary shape creation with a

single stroke- Automatic feature point detection- Modeless drawing- Natural feel

Metin Sezgin

Future work

• Integration with the new sketch recognition architecture

• Support for over-tracing

• Further investigation of sketching behavior to enhance/aid feature point detection and stroke classification