CSP Examples Tamara Berg CS 560 Artificial Intelligence Many slides throughout the course adapted from Svetlana Lazebnik, Dan Klein, Stuart Russell, Andrew.

CSP Examples

Tamara BergCS 560 Artificial Intelligence

Many slides throughout the course adapted from Svetlana Lazebnik, Dan Klein, Stuart Russell, Andrew Moore, Percy Liang, Luke Zettlemoyer

Announcements/Reminders

• HW1 due date extended to Sunday, Sept 13, 11:59pm on classroom server – submit one solution per group and email Ric (

[email protected]) with the location of your directory and names of your group members

Review from last class

Backtracking search

• In CSP’s, variable assignments are commutative– For example, [WA = red then NT = green] is the same as

[NT = green then WA = red]• We only need to consider assignments to a single variable at

each level (i.e., we fix the order of assignments)• Depth-first search for CSPs with single-variable assignments is

called backtracking search

Example

Example

Example

Example

Example

Red? Green? Blue?x x x

x

Backtracking search algorithm

Improving Backtracking Efficiency

• Ordering– How should we order variables for assignment?– How should we order values from the domains?

Backtracking search algorithm

Improving Backtracking Efficiency

• Ordering– How should we order variables for assignment?– How should we order values from the domains?

• Filtering– Can we detect inevitable failures early?

Early detection of failure

Apply inference to reduce the space of possible assignments and detect failure early

Early detection of failure:Forward checking

• Keep track of remaining legal values for unassigned variables• Terminate search when any variable has no legal values

• Simplest form of constraint propagation makes each pair of variables consistent:– X Y is consistent iff for every value of X there is some allowed value of Y

Arc consistency

Consistent!

• Simplest form of constraint propagation makes each pair of variables consistent:– X Y is consistent iff for every value of X there is some allowed value of Y

Arc consistency

• Simplest form of propagation makes each pair of variables consistent:– X Y is consistent iff for every value of X there is some allowed value of Y– When checking X Y, throw out any values of X for which there isn’t an

allowed value of Y

• If X loses a value, all pairs Z X need to be rechecked

Arc consistency

• Simplest form of propagation makes each pair of variables consistent:– X Y is consistent iff for every value of X there is some allowed value of Y– When checking X Y, throw out any values of X for which there isn’t an

allowed value of Y

• Arc consistency detects failure earlier than forward checking• Can be run before or after each assignment

Arc consistency

Improving Backtracking Efficiency

• Ordering– How should we order variables for assignment?– How should we order values from the domains?

• Filtering– Can we detect inevitable failures early?

• Structure– Can we exploit problem structure?

Tasmania and the mainland are separate structures.

We can solve them independently and then combine their solutions.

More generally, we can decompose the CSP into separate connected components (sub-problems of the CSP).

Then the solution is the union of the sub-problem solutions.

This can be much more efficient than solving the entire CSP as one problem.

Algorithm for tree-structured CSPs• Choose one variable as root, order variables from root to leaves

such that every node's parent precedes it in the ordering

http://cs188ai.wikia.com/wiki/Tree_Structure_CSPs

Algorithm for tree-structured CSPs• Choose one variable as root, order variables from root to leaves

such that every node's parent precedes it in the ordering• Backward removal phase: check arc consistency starting from the

rightmost node and going backwards

http://cs188ai.wikia.com/wiki/Tree_Structure_CSPs

Algorithm for tree-structured CSPs• Choose one variable as root, order variables from root to leaves

such that every node's parent precedes it in the ordering• Backward removal phase: check arc consistency starting from the

rightmost node and going backwards• Forward assignment phase: select an element from the domain of

each variable going left to right. We are guaranteed that there will be a valid assignment because each arc is arc consistent

http://cs188ai.wikia.com/wiki/Tree_Structure_CSPs

Alternatives to backtracking

Local Search Methods

• Local Search: Improve what you have until you can’t make it better

• Generally much faster and more memory efficient than back-tracking search (but not necessarily complete)

Local search for CSPs• Start with an initial assignment of variables to values • Allow states with unsatisfied constraints• Attempt to improve states by reassigning variable values• Hill-climbing search:

– In each iteration, randomly select any conflicted variable and choose value that violates the fewest constraints

– I.e., attempt to greedily minimize total number of violated constraints

h = number of conflicts

Local search for CSPs• Start with “complete” states, i.e., all variables assigned • Allow states with unsatisfied constraints• Attempt to improve states by reassigning variable values• Hill-climbing search:

– In each iteration, randomly select any conflicted variable and choose value that violates the fewest constraints

– I.e., attempt to greedily minimize total number of violated constraints– Problem: local minima

h = 1

Origin of CSPs

CSP in computer vision:Line drawing interpretation

An example polyhedron:

David Waltz, 1975

Desired output:

Goal: Label lines in drawing as convex (+), concave (-), or boundary (>).

How could we set this up as a CSP?

Variables?

Domains?

Constraints?

CSP in computer vision:Line drawing interpretation

Four vertex types:

Constraints imposed by each vertex type:

David Waltz, 1975

CSPs in recent research!

Inferring Temporal Order of Images from 3d Structure

Grant Schindler, Frank Dellaert, Sing Bing Kang

IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2007.

http://www.cc.gatech.edu/~phlosoft/

The World Changes Over TimeHow can we reason about time in structure from motion problems?

Ultimate Goal

Produce time-varying 3d models of cities from historical photographs

Requires: Building 3d modelsReasoning about time

Temporal Inference Problem1. When was each photograph taken?2. When did each building first appear?3. When was each building removed?

Set of Photographs:Set of Objects:

Buildings

Temporal Inference Problem

(a, b) Time Interval

t Time

Temporal Inference ProblemGiven observations Z and scene geometry X, what are the temporal parameters T?

Images

Structure from Motion

(x,y,z) Position

(x,y,z) Translation(y,p,r) Rotation f Focal Length

Structure

Cameras

Correspondences

3D

Overview

3D Images + TimeStructure + Time Intervals

1891

1934

2007

Buildings

1885 - 1953

1906 - 2009

1968 - 2009

4DCorrespondences

Images

Overview

3D Images + TimeStructure + Time Intervals

1891

1934

2007

Buildings

1885 - 1953

1906 - 2009

1968 - 2009

4DCorrespondences

Images

Overview

3D Images + TimeStructure + Time Intervals

1891

1934

2007

Buildings

1885 - 1953

1906 - 2009

1968 - 2009

4DCorrespondences

Images

Overview

3D Images + TimeStructure + Time Intervals

1891

1934

2007

Buildings

1885 - 1953

1906 - 2009

1968 - 2009

4DCorrespondences

Images

1937 2001 2009

3D ReconstructionImages 3D Point Cloud

1891

1934

20071885 - 1953 1906 - 2009 1968 - 2009

Structure from MotionBundler Software by Noah Snavely(Snavely et al SIGGRAPH 2006)

Scene Reconstruction• Automatically estimate

– position, orientation, and focal length of cameras

– 3D positions of feature points

Feature detection

Pairwisefeature matching

Incremental structure

from motion

Correspondence estimation

Input Photos

Feature detection

• Detect features points on objects [Lowe, IJCV 2004]

Pairwise feature matching

• Match feature points between each pair of images

Correspondence estimation

• Link up pairwise matches to form matches across several images

Image 1 Image 2 Image 3 Image 4

Structure from motion

Camera 1

Camera 2

Camera 3R1,t1

R2,t2

R3,t3

p1

p4

p3

p2

p5

p6

p7

minimize

f (R, T, P)

Incremental structure from motion

See: e.g. http://www.cs.cornell.edu/projects/bigsfm/ for more info

Grouping Points into Buildings3D Point Cloud 3D Buildings

Building Geometry1. Convex hull of each group2. Fit ground plane to camera centers3. Extend convex hulls to ground

1891

1934

20071885 - 1953 1906 - 2009 1968 - 2009

Point GroupingGroup points according to both: a. Distance < threshold in 3D b. Observed simultaneously in 1+ images

3D Reconstruction: Points vs. ObjectsLower Manhattan

454 images 83,860 points 960 Buildings1928-2010

Reasoning About Time:From Constraint Satisfaction…

• Key Idea: Visibility of 3D points constrains image ordering

• Variables? Domains? Constraints?

• Disadvantage: Only relative image ordering

Visibility Reasoning

• Goal: reorder images (columns) to have as few violations as possible

observed missing occluded

Columns: imagesRows: points

Violates constraints:

Satisfies constraints:

Temporal Ordering via CSP• Goal: reorder images (columns) to have as few violations as possible

• Backtracking search: intractable (n! solutions for n images)

• Local search: start with random ordering of columns, swap columns or groups of columns to reduce the number of conflicts

Results

. . .

Temporal Ordering via CSP• Goal: reorder images (columns) to have as few violations as possible• Local search: start with random ordering of columns, swap columns

or groups of columns to reduce the number of conflicts

• Can also reorder the rows to group together points that appear and disappear at the same time – that gives you buildings

Results: Time-varying 3d model

. . .

. . .

Results

Conclusions

• Authors demonstrated that constraint satisfaction problems provide a powerful framework in which to solve temporal ordering problems

• Presented the first known method for solving this ordering problem.

• Remaining challenges? – Dealing with noise in classification of points as

observed/missing/occluded.