Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 1 11.3.2008 Augmented Reality VU 1 Projective Geometry.

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung

Professor Horst Cerjak, 19.12.20051

11.3.2008Augmented Reality VU 1 Projective Geometry Axel Pinz

Augmented Reality

X

Y

Z

• Real scene, scene coordinates

C

xC

yC

zC

R, t

• Camera(s)

xV

yV

zV

• Visualization (screen, HMD)

R, t

• Real table

• Augmented plant




Example 1: ARToolkit




Example 2: Structure + Motion [Schweighofer]




Pinhole Camera

• “real” camera

image plane πi (u,v): z = -f

uv

zx

y“principal”point (u0,v0) “optical axis”

p(u,v)

P(x,y,z)

f

“focal length” f

• 2D projection 3D scene

• p(u,v) ↔ line of sight = viewing direction

P’(x’,y’,z’)

• “Pinhole” C … “center of projection”

C




Projective Geometry

π0 ... z = 0 πi ... z = f

x

z

yxy

uv

p(u,v)

P(x,y,z) P’

P’’

!!!

“projective” camera, “normalized” camera: f = 11 stationary camera 1 coordinate system (x,y,z)camera-centered coordinate system ≡ scene coordinate system

Only points in π0 are not projected to πi

(u0,v0)C




Projective Images: Examples, Properties

• Impression of depth in images• Parallel lines meet at infinity• “infinity” is projected to finite

location in the image• “horizon”• “points at infinity”, …

[Triggs and Mohr]




Projective Images: Scaling




Projective Images: Foreshortening




Projective Images: Parallel Lines Meet

[Sonka, Hlavac, Boyle]




Projective Geometry vs. Computer Vision

• points in π1

• straight lines of sight

• projective reconstruction

• geometry, precise

• known correspondences

• discrete pixels in πi

• sampling theorem• lens distortion, aperture, depth of field• “oriented” projective rec. “in front of camera”• inherently imprecise estimation, minimization• “outliers” robustness




Example: Stereo Reconstruction

PC1

C2• projective geometry• computer vision

P~




• A unified geometric + algebraic framework

• Point

• Line

Algebraic Projective Geometry (1)

Tyxy

xp 00

0

0 ,

dkxy 0 cbyax

1

0 0

1

y

x

x

c

b

a

lxly

x

c

b

a




• Duality point ↔ line

• Unified approach: projective n-space Pn

point (n+1) - vector

Algebraic Projective Geometry (2)

21 llp

1l

2l

21 ppl

1p

2p

Tnxxx 11 ,,




Homogeneous Coordinates in Pn

c

b

a

k

c

b

a

l ~

0 0)()(

0

kkcykbxka

cbyax

Equivalence class of vectors

0

0

03R forms P2 … “projective plane”

Homogeneous coordinates , but only 2 DoFHomogeneous coordinates , but only 2 DoF

inhomogeneousinhomogeneous

3

2

1

x

x

x

y

x




Equivalence Class of Vectors

Without further knowledge, such situationsWithout further knowledge, such situationscannot be distingushed !cannot be distingushed !

A further example: Equivalence ofA further example: Equivalence of a toy car, closeup shot, anda toy car, closeup shot, andreal car, distant shotreal car, distant shot




• Point

• Line

• “ideal” points treated like any point x3≠0

„Fluchtpunkte“

• “line at infinity” the plane’s “horizon”

The Projective Plane (1)

21 llx

21 xxl

intersection ofintersection ofparallel lines !parallel lines !

02

1

x

x

1

0

0

l





• Adding the ideal points to R2 leads to the projective plane P2

• Covers all homo-geneous coordinates

0

0

0

3

2

1

x

x

x

[Hartley+Zisserman][Hartley+Zisserman]





ππ

Image of the “horizon” of π,Image of the “horizon” of π,““line at infinity” of line at infinity” of ππ

vanishing point,vanishing point,„„Fluchtpunkt“ =Fluchtpunkt“ =Bild eines Bild eines „„Fernpunktes“Fernpunktes“

• Projective geometry can map infinitely far points / lines to finite onesProjective geometry can map infinitely far points / lines to finite ones• No difference between finite and infiniteNo difference between finite and infinite• e.g. hyperbola is e.g. hyperbola is one continuousone continuous conic conic




There is also Projective Space P3 …

• Points

• Planes

•

• Lines: 4 DoF

• Dual line L*:

Txxxx

x

x

x

x

p 4321

4

3

2

1

Tdcba

d

c

b

a

a

plane point 0 ap

21

2

1

21

:

, points 2

ppl

p

p

pp

T

T

L

22* 0 TLL

duality point duality point ↔ plane↔ plane duality duality LL ↔ ↔ L*L*




Projective Transformations in Pn

• “projective transformation” = “collineation” = “projectivity” = “homography” H

• Invertible mapping Pn →Pn

• „geradentreue“ Abbildung• (n+1) x (n+1) matrix• In P2 :

• H has (n+1)2-1 DoF, H is non-singular

line aon lie ,, line aon lie ,, 321321 xxxxxx

HHH

3

2

1

333231

232221

131211

3

2

1

'

'

'

'

x

x

x

hhh

hhh

hhh

x

x

x

x

x

H




Projective Transformations in P2

• Translation

• Rotation

• Scaling

• Any combination, e.g.

xxt

t

x

x TT

'

100

10

01

xxRR

'

100

0cossin

0sincos

xxs

s

y

x SS

'

100

00

00

xxx

MSRTSRTM '




A Remark on Conics

• 2nd degree equation in the plane

• Homog. coord:

• Conic C:

• Five DoF, 5 points define a conic

022 feydxcybxyax

0

/y / 233231

2221

21

3231

fxxexxdxcxxbxax

xxxxx

fed

ecb

dba

xxx T

2/2/

2/2/

2/2/

, 0 on CCC




Back to Homographies – Examples (1)

Mapping between planesMapping between planes

central projection may be expressed by x’=Hx






Removing projective distortionRemoving projective distortion










Transformation for Points, Lines, Conics

• Point

• Line

• Conic

xx

H'

11 , ' TTTT ll HHHH

1' CHHC T




A Hierarchy of Transformations / Geometries (1)

• Isometric / Euclidean

– Invariants: length, angle, area

• Similarity

– Invariants: ratios of length / areas, angle, parallel lines

yxTE ttt

, :DoF 3 , 10

R

H

yxTS ttts

, s, :DoF 4 , 10

R

H





• Affine:

–

– 6 DoF: 2 x scale λ1,λ2; 2 x rot. θ,Ф; 2 x translation

– Invariants: parallel lines, ratios of parallel lengths, ratios of areas

10

1002221

1211

Ty

x

A

ttaa

taa

AH

2

1

0

0 ),()()(

DDRRRA





• Projective:

– 8 DoF: 2 x scale λ1,λ2; 2 x rot. θ,Ф; 2 x translation;

2 x line at infinity– Invariant: Cross-ratio CR of 4 collinear points

333231

232221

131211

vv

t

hhh

hhh

hhh

TP

A

H

BCAD

CDABCR

.

.

A B C D





1002221

1211

y

x

taa

taa

1002221

1211

y

x

tsrsr

tsrsr

333231

232221

131211

hhh

hhh

hhh

1002221

1211

y

x

trr

trr

Projective8dof

Affine6dof

Similarity4dof

Euclidean3dof

In 2D, a square transforms to:In 2D, a square transforms to:





vTv

tAProjective15dof

Affine12dof

Similarity7dof

Euclidean6dof

10

tAT

10

tRT

s

10

tRT

In 3D, a cube transforms to:In 3D, a cube transforms to:




Stratification

In AR, we take perspective images,

but we require metric (Euclidean) reconstruction!

How? The stratification of 3D geometry [Pollefeys 2.2]




Stratification of 2D / 3D Geometry

• Many possibilities, many approaches

• Examples:– Known · directions– Known points, lines, planes at ∞– Known lengths in the scene– “IAC” (“self-calibration”)– Known camera intrinsics

Camera calibration + relative orientationMultiview geometry, structure+motion

unknownunknownscenesscenes




Stratification Examples (1)

• Known points, line at infinityv1 v2

l1

l2 l4

l3

l∞

21 vvl

211 llv 432 llv

perspectiveperspective affineaffine





• Known · directions

affineaffine metricmetric(similarity, unknown scale)(similarity, unknown scale)





• Known plane at infinity

perspectiveperspective affineaffine





• Known · directions

affineaffine metricmetric(similarity, unknown scale)(similarity, unknown scale)





• Known lengths

metric metric (similarity,(similarity,unknown scale)unknown scale)

metricmetric(Euclidean,(Euclidean,known scale)known scale)

[Pollefeys IJCV’99][Pollefeys IJCV’99]





• ARToolkit

perspectiveperspectivemetricmetric(Euclidean,(Euclidean,known scale)known scale)




“Video AR” is (rather) simple

• Known artificial targets / markers• Uncalibrated perspective camera

– But: collineation required– Problems when e.g. strong lens distortions

• Augmentation of the video frames• Examples

– Artoolkit– Kutulakos

• Can be related to scene coordinates, but requires “ground truth” for markers




ARToolkit Demo ISAR 2000

observer’s viewobserver’s view immersive viewimmersive view




Kutulakos’ “Calibration-Free AR” [IEEE Trans. Visualization and Graphics 1998]




Field Maintenance Support [ARVIKA]




Scene Structure + Camera Motion(the harder, but more general approach to AR)

• Many possible approaches• Monocular, calibrated, known “natural”

landmarks [Ribo]

• Stereo, calibrated [Schweighofer]

• Monocular, calibrated [Murray]

• Monocular, uncalibrated [Pollefeys]

– not (yet?) in real time !

calibration !calibration !

unknownunknownscene, scene, unknownunknown “ “natural”natural”landmarkslandmarks

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 1 11.3.2008 Augmented Reality VU 1 Projective Geometry.

Documents

projective camera

augmented reality vu

projective plane p

scaling slide

foreshortening slide

artoolkit slide

boyle slide

mohr slide