-
Polarimetric Three-View Geometry
Lixiong Chen1, Yinqiang Zheng1, Art Subpa-asa2, and Imari
Sato1
1 National Institute of Informatics2 Tokyo Institute of
Technology
{lchen,yqzheng,imarik}@nii.ac.jp, [email protected]
Abstract. This paper theorizes the connection between
polarizationand three-view geometry. It presents a ubiquitous
polarization-inducedconstraint that regulates the relative pose of
a system of three cameras.We demonstrate that, in a multi-view
system, the polarization phaseobtained for a surface point is
induced from one of the two pencils ofplanes: one by specular
reflections with its axis aligned with the inci-dent light; one by
diffusive reflections with its axis aligned with the sur-face
normal. Differing from the traditional three-view geometry, we
showthat this constraint directly encodes camera rotation and
projection, andis independent of camera translation. In theory, six
polarized diffusivepoint-point-point correspondences suffice to
determine the camera rota-tions. In practise, a cross-validation
mechanism using correspondencesof specularites can effectively
resolve the ambiguities caused by mixedpolarization. The
experiments on real world scenes validate our proposedtheory.
1 Introduction
When an unpolarized incident light is reflected by a dielectric
surface, it be-comes polarized and the phase of its polarization is
characterized by the planeof incidence. This process can be
observed by a rotatable polarizer mounted infront of a camera that
captures sinusoidally varying pixel-wise radiance, wherethe
readings arising from specular reflection exhibits a π2 phase shift
relative tothe readings from diffusive reflections. In both
phenomena, the phase shift ofthe sinusoids indicates the azimuthal
orientation of the surface normal, and itselevation angle is
evaluated by the reflection coefficients [13]. Essentially,
polari-metric measurements impose a linear constraint on surface
normals [43], whichis useful for shape estimations under
orthographic projection.
We note that, the relative phase of polarimetric measurements by
a tripletof cameras alone encodes sufficient information to
describe the relative poseof these cameras. As illustrated in
Figure 1, characterizing general surface re-flectance is usually
the plane of incidence formed by the incident light and theline of
sight. Geometrically these planes are organized in a way to
representreflection/refraction under two common scenarios: (1)
direct surface reflectiondue to a directional light which displays
specularities; (2) diffusive reflectionsdue to subsurface
scattering that render the surface’s own property. In the first
-
2 L. Chen et al.
scenario all planes of incidence intersect on a set of parallel
lines aligned with in-cident light, and in the second scenario
other planes exist to intersect on the linepassing through the
surface normal. These pencils of planes impose a
geometricconstraint on the plane orientations explicitly through
the relative rotations ofthe cameras. Specifically, three planes
(e.g. camera poses) uniquely specify theline of intersection.
In the inverse domain, each pencil of planes is represented by a
3-by-3 rank-2matrix, so accordingly six instances of such matrix
are sufficient to determinethe camera rotations. However, the
number of possible constructions of thesematrices grows
exponentially due to the π2 -ambiguity caused by the mixed
po-larizations [3], hence directly solving the minimal problem is
numerically pro-hibitive. Fortunately, since often the ambiguities
occur only when specularitiesare present, the π2 phase shift can be
effectively leveraged if we only defer theiruse for verifications,
but not directly for estimations. Specifically, since
con-structions using incident light are easy to establish, we
obtain the correspondingmatrices and make additional three attempts
for each instance with π2 difference,as doing so effectively
cross-validates the co-existing constructions induced bysurface
normal.
To sum up, by estimating the relative rotation of a triple of
cameras, in thispaper we elucidate a fundamental connection between
polarization and three-view geometry. In particular, our
contributions are as follows:
1. Using microfacet theory, we identify and theorize the
ubiquitous existenceof two types of a pencil of planes induced by
polarizations from generalreflectance.
2. We formulate a geometric constraints using the induced pencil
of planes,under which we show that in a triplet of cameras
polarimetric informationcan be leveraged to extract the cameras’
rotations from its translation.
3. We use experiment to validate our theories, in particular, we
propose to usecorrespondences of specular points to address mixed
polarizations.
The rest of this paper is organized as follows: Section 2
overviews the relatedwork, Section 3 explains the polarization from
general reflectance using micro-faect theory, and by examining the
measured relative phase we illustrate theexistence of two types of
polarization from reflection. Section 4 extends ourformulation to
three-view geometry, revealing how camera rotation can be de-coded
from polarimetric information. Our experiments on real world scenes
aredescribed in Section 5. Section 6 discuss our plan for the
future work and con-cludes this paper.
2 Related Works
This work is related to two lines of research: one applies
polarization as a visualcue for shape and depth estimation, the
other formulates three-view geometryusing trifocal tensors.
-
Polarimetric Three-View Geometry 3
(a) polarization by surface nor-mal
(b) polarization by light
Fig. 1: This paper examines two types of polarization-induced
geometrical config-urations. One is a pencil of planes induced by
diffusive reflection which encodesthe information about the surface
normal, the other is induced by specular re-flection and it encodes
the light direction
2.1 Shape and Depth Estimation from Polarization
Following the Fresnel equations [13], ideal mirror reflection
allow the azimuthangle and zenith angle of the normal of the mirror
surface to be evaluated. Thisphysical model can be generalized to
more realistic cases where relaxed assump-tions are made for the
controlled light, the camera pose, and the reflectanceproperty of
the surface. Correctly identifying the orientation of the plane of
in-cidence among multiple ambiguous interpretations is a common
challenge thatmany applications face to address.
Direct shape estimation based on polarization under single view
[28, 5] forphotometric stereo often targets on a surface of known
reflectance under con-trolled illumination [9, 32]. For example, it
is intuitive to recover the shape of aspecular surface because
specularities always display strong polarization effect[35]. It
also reasonable to leverage polarization observed from transparent
objects[26, 25], the objects covered by metallic surface [29], or
those made of scatteringmedium [30]. It has been demonstrated that
diffusive reflection can carry po-larization signals due to
subsurface scattering [3]. Shading can be integrated toenhance the
estimations [23], and in the presence of mixed polarization,
labelingdiffusive and specular reflectance [31] turns out to be
useful in some applications[38]. Additionally, designed
illumination pattern can also be applied to enrichthe polarization
effect [1].
Another typical example of applying the polarimetric cues is to
fuse themto constrain the depth map obtained using other means. The
depth signals caneither be obtained physically [17, 18] or
geometrically inferred [37, 40]. The un-derlying assumptions made
is that the surfaces tend to be smooth or can beeasily
regulated.
A multi-camera setup produce a richer set shape cues [2, 4],
reduce the oc-currences of ambiguous measurements, and avoid the
formulation involving therefractive index, which is dealt directly
in some cases [15, 16]. Polarimetric cuescan facilitate the dense
two-view correspondence over specular surfaces [6]. In a
-
4 L. Chen et al.
standard structure-from-motion setting, camera poses are first
estimated usingthe classical approaches before [27, 8] polarimetric
information is applied. Recentwork also integrates it into SLAM
[44]. In our work, we show that polarimetricinformation can also be
applied to retrieve camera pose, which to our knowledgeis the first
demonstration of its usefulness in the related field.
2.2 Three-View Geometry
Analogous to the role of fundamental matrix in two-view
geometry, the three-view geometry is characterized by the trifocal
tensor that relates point or linecorrespondences across three views
[10]. From a historical viewpoint, the termtrifocal tensor
originated from the seminal studies [36, 11] on the trilinear
con-straints for three uncalibrated views, although their
counterpart for line tripletsin three calibrated views [42]
appeared much earlier. The 3× 3× 3 tensor has 27elements, yet the
degree of freedom is 18 only in the projective case, which mean-s
that these elements should comply with 9 constraints. This
naturally arousesthe problem of minimal parametrization, which has
been widely addressed inthe literature [39, 7, 33, 22].
To estimate the trifocal tensor in projective reconstruction
requires at least 6point triplets, for which Quan [34] proposed an
effective method. On the contrary,no less than 9 line triplets are
required for this estimation, for which the state-of-the-art solver
in [21] is still too huge to be practical. Therefore, it is
commonto use a linear estimation method using 13 or more line
correspondences [12],and refine the result through iterative
nonlinear optimization. Trifocal tensorestimation in the calibrated
case is involved as well, because of the presenceof two rotations.
A specialized minimal solver is presented in [21] for cameramotions
with given rotation axis. Very recently, Martyushev [24]
characterizedthe constraints on general calibrated trifocal tensor,
which include 99 quinticpolynomial equations. Kileel [19] studied
the minimal problems for calibratedtrifocal variety and reported
some numerical results by using the homotopycontinuation method.
Since the computation is prohibitively slow, people tendto solve
the essential matrix arising from two-view geometry instead.
3 Polarimetric Reflectance under Single View
Polarization arises when an incident light propagates through a
surface betweentwo mediums of different refractive indices. Fresnel
equations describe an idealphysical model that only considers
single bounce surface reflection from direc-tional light. As
illustrated in Figure 2a, the light is thought of as a linear
su-perposition of two independent EM waves: one whose oscillation
is in the planeof incidence Π⊥(n) perpendicular to the surface of
normal n, one oscillates inthe plane Π‖(n) parallel to the same
surface. As an unpolarized light impingeson the surface, its
propagation bifurcates: one branch is immediately reflectedoff from
the surface, the other refracts through the surface. The two wave
com-ponents share their path, but how they allocate the power upon
bifurcation is
-
Polarimetric Three-View Geometry 5
(a) (b)
Fig. 2: The Fresnel equation explains how an unpolarized
incident light becomespolarized through mirror reflection. (a) When
reflection and refraction take placeunder directional light, the
polarization phase indicates the orientation of theplane of
incidence, and there will be no light traveling outside it. (b)
Insidethe plane of incidence, coplanar propagations along multiple
paths must exhibitidentical phase, hence it has to be
differentiated by wave magnitude.
opposite. Along the light path after bifurcation, one wave
component alwaysoutpowers the other, and the magnitude of power
discrepancy is measured bydegree of polarization. In the plane Π
where the polarizer is located, theangular distance between the
peaks of a wave component along different pathsis measured by the
relative polarization phase. By conservation of energyand
orthogonality, we establish the following:
Proposition 1 At a dielectric surface boundary, any pair of
reflected or re-fracted light inside parallel incident planes is
always in phase (i.e. 0 relativepolarization phase), and any pair
of reflected and refracted light is always out ofphase (i.e. π2
relative polarization phase) (i.e. out of phase
3).
The phenomena described in Proposition 1 indicates that the
relative polariza-tion phase is shared by co-plannar
reflections/refractions, as indicated in Fig-ure 2b. Since incident
plane contains the information about both of the surfaceand the
incident light, as opposed to the existing literature that
elaborates onthe degree of polarization, in the following we
investigate the connection betweenthe polarization phase and some
important geometrical properties pertaining toview, scene and
light.
3.1 Polarization Defined by Directional Light
Our investigation starts with a formulation with directional
light. We modelthe surface using a typical microfaceted setting
[41], namely a subset of mirror-like microfacets are selected by
the unit vector h bisecting the line of sight
3 degree of polarization varies periodically with periodicity of
π
-
6 L. Chen et al.
(a) (b)
Fig. 3: Polarization of general reflectance over a rough surface
can be understoodthrough microfacet configuration. A unique
configuration made by the line ofsight v and the incident light l
will only activate the microfacets aligning withthe bisector h. (a)
Each microfacet acts as a tiny mirror so that its reflectionfollows
Fresnel equation. (b) When light carrying constant power impinges
fromall directions, the aggregated polarization effect observed can
be approximatedas if it is measured from a mirror with the same
orientation, hence the readingsindicate the surface normal.
vector v and light vector l to produce a specular reflection.
The spatially varyingreflection depends on the effective visible
area A(h) [14] formed by the selectedfacets according to the
microfacet distribution function. As depicted in Figure3a, specular
reflection is solely determined by the direction of the light but
notthe scene structure. Essentially, the observation is the outcome
of a structuredefined by Π⊥(h) and Π‖(h) whose properties are
summarized by Proposition1. Therefore we arrive at the
following:
Proposition 2 Under directional light, the relative polarization
phase due togeneral surface reflection is indicated by the
projection of the incident planeformed by l and v onto the
polarizer.
We can experimentally verify this fact using two observations
presented in Figure4a and 4b: when the line of sights tend to be
parallel, the relative phase ofpolarization is in phase and
apparently independent of surface orientation, butit can be
affected by perspective projection.
Moreover, let I⊥(h) and I‖(h) be the power of the two orthogonal
wavecomponents confined in Π⊥(h) and Π‖(h) respectively, then a
polarizer withrotation w in its own coordinates reads:
I(w,h) = A(h)I⊥(h) cos2 θ +A(h)I‖(h) sin
2 θ (1)
with θ denotes the angle made between w and the projected line
from Π⊥(h)to the polarizer. It is worth noting that Equation 1 is
the microfacet version ofthe expression for the sinusoidal curve
that has been widely analyzed in otherliteratures. For surface
reflections, I(w,h) vanishes only when v and h makethe Brewster’s
angle. Hence, the polarizer essentially detects the configurationof
Π‖(h) and Π⊥(h) for a specific h.
-
Polarimetric Three-View Geometry 7
3.2 Polarization Defined by Surface
While how a directional light becomes polarized through
reflection depends onits incident angle, under environment light of
uniform power, the collective be-havior of polarization reflects
the surface geometry. For reflectance received fromenvironment map
Ω+, by Equation 1 the radiance perceived by polarizer withrotation
w is defined as:
I(w) =
∫
Ω+
A(h)I⊥(h) cos2 θ +A(h)I‖(h) sin
2 θ dh
=
∫ π
2
−π2
cos2 θ
∫
Π(φ)
A(h)I⊥(h)dh+ sin2 θ
∫
Π(φ)
A(h)I‖(h)dhdφ (2)
=
∫ π
2
−π2
F (φ,w)dφ
where h ∈ Π(φ) lies in a plane that is orthogonal to the image
plane Π, cre-ating an aggregation of coplanar reflection as
described by Proposition 1 anddemonstrated in Figure 3b. Since F
(φ) exhibits an identical structure to Equa-tion 1, I(w) can be
understood as a composition of a set of distinctive
sinusoidalcurves sharing some specific φ. In other words, F (φ,w) =
Imin(φ) + (Imax(φ)−Imin(φ)) cos(θ+ φ), where Imin and Imax are
determined by
∫
Π(φ)A(h)I⊥(h)dh
and∫
Π(φ)A(h)I‖(h)dh. Here evaluating their exact quantities is
unnecessary.
If A(h) is derived from a material displaying isotropic
reflectance, v avoidsthe grazing incidence (i.e. n⊺v ≫ 0), then the
shadowing effect becomes minor(i.e. equal to 1), and as a result
A(h) becomes rotationally invariant about n (i.e.A(h1) = A(h2)
given that n
⊺h1 = n⊺h2). By symmetry about φ = 0 we have
Imax(φ) = Imax(−φ) and Imin(φ) = Imin(−φ) under the environment
light ofuniform power. Furthermore, F (φ,w)+F (−φ,w) = 2Imin(φ)+2
cosφ(Imax(φ)−Imin(φ)) cos θ, which is in phase with F (φ = 0).
Therefore, Equation 2 leads toI(w) = C1(v,n)F (φ = 0,w) with
C1(v,n) being some constant.
In practise, A(h) peaks when h = n. Also, Fresnel equation
implies thatat grazing incidence the mirror reflection becomes
dominant, meaning the lightthat leads to h⊺v → 0 and h → n
contributes the most to the actual reflectance.Therefore, when v is
set at the grazing angle, I(w) = C2(v,n)F (φ = 0,w) alsoserves as a
good approximation for Equation 2. Combining these two scenarios,we
summarize the following:
Proposition 3 Under environment light of constant power, the
relative polar-ization phase of general surface reflection is
indicated by the projection of theplane formed by n and v onto the
polarizer.
3.3 Mixed Polarization with Diffusive Reflection
In practise, diffusive reflection due to subsurface scattering
is usually observedin tandem with surface reflection. Because
refracted light tends to depolarize
-
8 L. Chen et al.
isotropically as it is scattered by the microstructure
underneath the surface, aportion of it has a chance to refract back
after several bounces and rejoins thepropagation of directly
reflected light [3]. This process to generate diffusive reflec-tion
can be thought of a byproduct of direct surface reflection by the
environmentmap of constant power Ω− covering the lower hemisphere.
By proposition 1 andproposition 3 we derive the following for the
observation made in Ω+:
Proposition 4 The relative phase of general diffusive reflection
is determinedby the projection of the plane formed by n and v onto
the polarizer, and it differsin phase from the direct surface
reflection by π2 .
This endorses the finding claimed in [3, 8]. This fact together
with Proposition 3can be experimentally verified and the results
are demonstrated in Figure 4c.
To sum up, under single view the relative polarization phase
measured for aspecific scene point might be led by two types of
phenomena: the specular reflec-tions encoding the incident light or
the diffusive reflections encoding the surfacenormal. It is worth
noting that the conclusions made heretofore is independentof the
settings for camera. Section 4 shows that by unifying the
polarizationphase obtained from different views, one can retrieve
the relative rotations ofthe cameras.
(a) (b) (c)
(d) (e) (f)
Fig. 4: The relative phase measured under single view with
various light-view-geometry configuration. It can be seen that
specular reflection is dependentonly on view and light, while
diffusive reflection depends on the geometry ofthe scene. (a)(d)
orthographic specular reflection displays in phase
polarization.(b)(e) perspective specular reflection displays
slightly out-of-phase polarization.(c)(f) polarization phase shift
due to diffusive reflections indicates the geometryof the
scene.
-
Polarimetric Three-View Geometry 9
4 Polarimetric Geometry under Three Views
The relative pose between the camera and a scene point is
regulated by two typesof planes: (1) those formed by v and l
(Section 3.1) and (2) those formed by vand n (Section 3.2 and 3.3).
Accordingly, in a multi-view setup, for each pointthere exist two
clusters of planes, one belongs a type. Inside each cluster,
theorientation of the plane in the camera’s local coordinates is
represented by thedetected relative polarization phase. We show
that, using a static scene understatic illumination, the
polarization phases captured from three distinctive viewsavail us
the relative pose of a the cameras.
4.1 Formulation
We setup a system of cameras indexed by j with optical center
denoted by oj .Their poses are described by rotation matrices Rj
together with the correspond-ing translation vectors tj . Each
camera pose has six degrees of freedom, withthree of them
parameterizing R. As indicated in Figure 5, Let Si denote a
scenepoint indexed by i. From Section 3 we know that linking each
point Si to cameraj is a vector hi,j that represents the projection
of h onto the image plane Πcentered at oj . h is obtained by
fitting w to Equation 1, which does not involveprojection. Let nij
denote the normal of the induced plane of incidence Πij andvij the
line of the sight, and according to the reflectance type we either
havenij = ni × vij or nij = l × vij . Moreover, there exists a
matrix, N i for scenepoint Si as:
N i =[
ni,1 R2ni,2 . . .Rjni,j . . .]
(3)
where we let R1 = I. Correspondingly, another matrix , N l, can
also be con-structed for directional light l. By definition we
have:
N i = [ni]×[
vi,1 R2vi,2 . . .Rjvi,j . . .]
(4)
andN l = [l]×
[
vi(1),1 R2vi(2),2 . . .Rjvi(j),j . . .]
(5)
where i(j) indexes the position of the floating specularity
observed from view j,and [·]× is the matrix representation for
cross product, whose rank is always 2.Therefore, the rank of both N
i and N l is also 2.
Equation 4 and 5 indicate that, the aforementioned cluster of
plane {Π}ijare two pencils of planes: one has axis aligned with ni,
and the other has axispassing through l. The difference is that Ni
represents a pencil of planes whosemembers physically coincide with
Πij , while N(l) indicates a pencil of planesthat contains
translated Πij , as depicted in Figure 5. In both cases the
rank-2constraints hold, hence our derivation can be summarized as
follows:
Proposition 5 In a multi-view system with one dominant
directional light, thepolarization displayed by a scene point may
induce one of two pencils of planes,one has its axis aligned with
the propagation of the directional light, and theother has its axis
passing through the surface normal.
-
10 L. Chen et al.
(a) structure pencils of planes (b) perspective three view
geometry
Fig. 5: Construction for three-view geometry. The structure
represented by thetwo figures are identical given that the incident
light are parallel, but physicallyincidence planes induced by light
are not necessary intersect on the single line
Since [l]× denotes light direction, [ni]× represents scene
structure, and vij isrepresented pixel location, Equation 4 and 5
effectively decouple camera trans-lation from the camera rotation
and camera model. So, polarimetric informationis highly useful for
rotation estimation.
4.2 From Three-View Polarization to Camera Rotation
For camera pose estimation, the rank-2 constraint imposed on N i
and N l iscritical. It allows us to set up a theoretical
formulation for the correspondingminimal system and then extend it
into a relaxed least square setup. More im-portantly, leveraging
both N l and N l can resolve ambiguity caused by mixedpolarizations
effectively.
An Extended Least Square Solver In the minimal case N i and N l
are two3-by-3 matrices (i.e. j ∈ {1, 2, 3}) to be determined
through R2 and R3, whichin our formulation are expressed by two
unit-norm 4-by-1 vectors q2 and q3 inquaternions respectively. Each
vector contains three unknowns, so six points toform six pencils of
planes of unique axes can completely determine R2 and R3.In
particular, we establish a system of six equations of 4-th order
polynomials:detN i = 0 with an additional constraint detN l = 0 (1
≤ i ≤ 6) to resolve theπ2 phase ambiguity caused by mixed
polarization.
As mentioned, directly solving the minimal problem using 6
points is compu-tationally challenging. A simple instance we
created for off-line evaluation showsthat the correct solution is
buried among 4252 candidates in the complex do-main. Aside from
applying additional assumptions [21], for our setup we proposeto
directly apply the non-linear least square solver that takes few
more points.We believe this is feasible for two reasons: (1) we
only need a sparse set of ro-bust correspondences to define camera
pose; (2) polarization measurements aresusceptible to noise,
relaxed formulation should strengthen our esitmation.
-
Polarimetric Three-View Geometry 11
Resolving Mixed Polarization In the presence of specularity, π2
-ambiguitydue to mixed polarizations observed from three views may
result in each nij hav-ing 8 possible interpretations. This
combination makes even a minimal systemprohibitively large to solve
(68 = 1679616). Ordering the strength of specularitywill reduce the
number of combinations (64 = 1296), but this reduced set isstill
far from being feasible. On the other hand, under general
reflectance withcomplex scene structure, specularities often appear
but distribute sparsely inspace. In other words, if majority of
point correspondences diffusive-diffusive-diffusive, few
specular-specular-specular may be excluded through intensity
pro-filing. However, there is a chance that ideal diffusive
correspondences being mis-taken as specular ones are excluded. In
our case we can construct a hypotheticalN l using the estimated
result to verify the result. If the estimation is accurate,the
resulting matrix should also be rank-deficient. Such consistency
motivatesus to design a solution consisting of two subroutines with
one to address the π2ambiguity caused by specularities produced by
a directional light:
selecting diffusion-only correspondences Excluding the
correspondences in-volving plausible specularities by intensity
profiling (i.e. the brightest pixelsin the scene). Applying the
remaining correspondences to create instancesof N i, and solve for
min(
∑
i detN i)2.
disambiguating using specularities Including the plausible
specularities toconstruct a hypothetical N l. If the construction
is valid, it has to be rankdeficient. Otherwise, flip the input by
π2 to detect a minimum determinant.This can be achieved after 3
attempts. Then make it a input to the estimator.
The above procedure proceeds iteratively until no flipping can
help improve theresults.
Essentially, N l serves as a robust constraint to cross-validate
the consistencyover all observations. This design draws strong
analogy to the RANSAC-basedmethods for feature correspondences.
Designing a better framework integratingboth is left as part of our
future work.
4.3 Illumination, Structure and Camera Calibration
Knowledge about N i and N l can be further applied to retrieve
the lines carrysurface normal, the direction of the light and the
camera’s focal length. Underorthographic or weak perspective
projections, Equation 4 and 5 can bereduced to:
Ni = [ni]×[
I R2 . . .Rj . . .]
v (6)
andNl = [l]×
[
I R2 . . .Rj . . .]
v (7)
respectively, where under orthographic projection v = [0, 0, 1]
and under weakperspective projection |v| is assumed to be an
unknown constant (i.e. indepen-dent of the actual scene structure).
Orthographic projection only considers rota-tion, and it is a
common assumption made for normal estimation in the existing
-
12 L. Chen et al.
literatures. Weak perspective projections, on the other hand,
additionally con-sider camera translation over a unknown spherical
surface. In both situations onecan recover surface normal according
to Equation 6, and light direction accord-ing to Equation 7.
Perspective projection with focal length fj , vij = (xi, yi, fj)and
the optical axis passes through the square image center yield a
system ofquadratic equation in terms of ni and fj by Equation
4.
4.4 Comparison with Trifocal Tensor
With P j = [P j,1:3|P j,4] : S → sj being the projection
operator projectingS onto the image plane Πj , we are able to link
the formulation presented inSection 4.1 to trifocal tensor
[10]:
M =
[
P⊺
1,1:3h1 P⊺
2,1:3h2 P⊺
3,1:3h3
P⊺
1,4h1 P⊺
2,4h2 P⊺
3,4h3
]
(8)
where hj is the line projected onto the image planesΠj . M is a
4-by-3 matrix andrank(M) = 2. Equation 4, 5 and Equation 8 display
similar algebraic propertiesand exhibit the following connections:
(1) hi,j arises naturally from polarization,so line correspondence
is achieved without a line detector marking points alonga visible
line for correspondence. (2) N i in Equation 4 occupies the first
threerows of M subject to linear scale, so any algorithms designed
to address trifocaltensor can be tailored for polarization. (3) The
fourth row of the trifocal tensorencodes the camera translation.
Therefore, we see that the relative polarizationphase essentially
serves as a useful cue for camera rotations.
5 Experiments
In order to verify our theory under the proper illumination
setup, we requireat most one strong and directional to be present.
In our case this can be thelight mounted on the ceiling. A linear
polarizer is embedded inside a motorizedrotator, and it is mounted
in front of a grey scale camera, which we calibrateaccording to
[45]. In our experiment, we use 11 distinctive exposures to
obtainthe HDR images for each scene to reduce saturation. Also, for
each exposurewe average the result multiple times in order to
reduce the thermal noise of thedevice. We perform verifications and
pose estimations in separate experiments.In each scene,
checkerboards are also included to obtain the ground truth.
5.1 Verification
We use two separate scene to verify the existence of rank
deficient matrices, N iand N l, respectively. We use “dice” to
setup the scene for diffusive reflectionsand “ball” to produce
specular reflections. Specifically, in “dice” we manuallyselect 20
anchor correspondences and then populate the correspondences
usingtheir neighboring pixels. We evaluate the statistics of the
singular values of the
-
Polarimetric Three-View Geometry 13
(a) view 1 (b) view 2 (c) view 3 (d) (e)
Fig. 6: Verification experiment for diffusive reflections. (d):
the statistics of thesingular values obtained from the sampled
instances. (e): a plot for intensityvariation of a good
correspondence.
(a) view 1 (b) view 2 (c) view 3 (d) (e)
Fig. 7: Verification experiment for specular reflections. (d):
the statistics of thesingular values obtained from the sampled
instances. (e): a plot for intensityvariation of a good
correspondence.
obtained matrices. From Figure 6 we observe that the smallest
singular valuemaintain to be significantly lower than the largest
singular value, indicating thatthe matrices indeed tend to be rank
deficient in practise.
For specular settings, we select 30 samples from the brightest
pixels andconstruct N l through random matching. The statistics of
singular values showthat it is also highly rank deficient because
the smallest singular value on averagealmost vanishes compared with
the largest singular value. Also, in both scenariosthe intensity
variations of good correspondences display clean sinusoidal
curveswith apparent phase shift, and their magnitudes do not affect
the structure ofour proposed structures.
5.2 Estimation
We set up a real-world scene to showcase our solution, and its
estimation resultsare visualized in Figure 8. Our goal is to
estimate the rotations, and the due tothe space limit our
configuration leads to orthographic projection. The
resultedrotation matrices are evaluated relative to its ground
truth. Here R12 indicatesthe relative rotation from view 2 to view
1: R12 = (0.9977, 0.9915, 0.9892), R12 =(0.9855, 0.9797, 0.9652)
which are intuitively reasonable.
The estimation accuracy are mainly degraded by two factors: (1)
the measure-ment noise that are commonly observed for polarization
measurements, whichoccurs often from diffusive reflections and cast
cast shadows; (2) the correspon-dence might not accurate. In the
experiment we also manually include some
-
14 L. Chen et al.
(a) view 1 (b) view 2 (c) view 3 (d) R12 (e) R13
Fig. 8: An example for estimating the camera pose using
polarimetric readings.
plausible correspondences inside the textureless region. Since
the synergy ofthese two factors amplifies our estimation error, an
effective solution to thisissue is under our investigation.
6 Conclusions and Future Work
In conclusion, in this paper we establish the theoretical
connection between po-larization and three-view geometry, which
leads to an example of polarization-enabled estimation on camera
poses. In particular, guided by the microfacettheory and the
classical Fresnel equations, we experimentally verify the
ubiq-uitous existence of the two types of pencils of planes derived
from polarizationphase shift, where one is induced by the direct
surface reflections and the otherby the diffusive reflections due
to subsurface scattering. Our formulation showsthat a rotatable
linear polarizer can extract the relative rotation of a camerafrom
its translation. Also, using pencil of planes induced by light, the
specularcorrespondences cross-validate the estimation obtained from
diffusive correspon-dences with fixed number of steps, which we
consider an effective strategy to re-solve ambiguities caused by
mixed polarizations. Our experiment on real worldscene validates
our theory and produce desirable results.
However, it is not hard to see that our experiment is still
preliminary. Be-cause polarization measurements are vulnerable to
noises, whose effect amplifiesunder uncontrolled illumination. In
particular, polarization by diffusive reflec-tions delivers less
stable observations than specular reflections do due to thethermal
noise of the device. On the other hand, however, diffusive
reflection-s due to subsurface scattering usually carry the dense
features for traditionalstereo correspondences. These features are
also the key reasons that RANSAC-based approaches are resilient to
noise. Since our strategy for disambiguation ofmixed polarization
described in Section 4.2 operates in a similar manner, it
isreasonable to put both parts into a unified framework. Comparing
with fusingpolarimetric information structure reconstruction [8],
our work showcase thatpolarization can be directly used to extract
some underneath geometric proper-ties about the camera and the
scene, which also draws certain analogies to thework of traditional
setup [20]. Therefore, exploring the geometric properties em-bedded
inside polarization and integrating them into the traditional
frameworkwill be a part of our future work.
-
Polarimetric Three-View Geometry 15
References
1. Atkinson, G.A.: Two-source surface reconstruction using
polarisation. In: Scandi-navian Conference on Image Analysis. pp.
123–135. Springer (2017)
2. Atkinson, G.A., Hancock, E.R.: Multi-view surface
reconstruction using polariza-tion. In: Computer Vision, 2005. ICCV
2005. Tenth IEEE International Conferenceon. vol. 1, pp. 309–316.
IEEE (2005)
3. Atkinson, G.A., Hancock, E.R.: Recovery of surface
orientation from diffuse polar-ization. IEEE transactions on image
processing 15(6), 1653–1664 (2006)
4. Atkinson, G.A., Hancock, E.R.: Shape estimation using
polarization and shadingfrom two views. IEEE transactions on
pattern analysis and machine intelligence29(11) (2007)
5. Atkinson, G.A., Hancock, E.R.: Surface reconstruction using
polarization and pho-tometric stereo. In: International Conference
on Computer Analysis of Images andPatterns. pp. 466–473. Springer
(2007)
6. Berger, K., Voorhies, R., Matthies, L.H.: Depth from stereo
polarization in spec-ular scenes for urban robotics. In: Robotics
and Automation (ICRA), 2017 IEEEInternational Conference on. pp.
1966–1973. IEEE (2017)
7. Canterakis, N.: A minimal set of constraints for the trifocal
tensor. In: EuropeanConference on Computer Vision. pp. 84–99.
Springer (2000)
8. Cui, Z., Gu, J., Shi, B., Tan, P., Kautz, J.: Polarimetric
multi-view stereo. In:Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition.pp. 1558–1567 (2017)
9. Drbohlav, O., Sara, R.: Unambiguous determination of shape
from photometricstereo with unknown light sources. In: Computer
Vision, 2001. ICCV 2001. Pro-ceedings. Eighth IEEE International
Conference on. vol. 1, pp. 581–586. IEEE(2001)
10. Hartley, R., Zisserman, A.: Multiple view geometry in
computer vision. Cambridgeuniversity press (2003)
11. Hartley, R.I.: Lines and points in three views and the
trifocal tensor. InternationalJournal of Computer Vision 22(2),
125–140 (1997)
12. Hartley, R.I., et al.: Projective reconstruction from line
correspondences. In:CVPR. pp. 903–907 (1994)
13. Hecht, E.: Optics. Pearson Education (2016)14. Heitz, E.:
Understanding the masking-shadowing function in
microfacet-based
brdfs. Journal of Computer Graphics Techniques 3(2), 32–91
(2014)15. Huynh, C.P., Robles-Kelly, A., Hancock, E.: Shape and
refractive index recovery
from single-view polarisation images. In: Computer Vision and
Pattern Recognition(CVPR), 2010 IEEE Conference on. pp. 1229–1236.
IEEE (2010)
16. Huynh, C.P., Robles-Kelly, A., Hancock, E.R.: Shape and
refractive index fromsingle-view spectro-polarimetric images.
International journal of computer vision101(1), 64–94 (2013)
17. Kadambi, A., Taamazyan, V., Shi, B., Raskar, R.: Polarized
3d: High-quality depthsensing with polarization cues. In:
Proceedings of the IEEE International Confer-ence on Computer
Vision. pp. 3370–3378 (2015)
18. Kadambi, A., Taamazyan, V., Shi, B., Raskar, R.: Depth
sensing using geometri-cally constrained polarization normals.
International Journal of Computer Vision125(1-3), 34–51 (2017)
19. Kileel, J.: Minimal problems for the calibrated trifocal
variety. SIAM Journal onApplied Algebra and Geometry 1(1), 575–598
(2017)
-
16 L. Chen et al.
20. Kneip, L., Siegwart, R., Pollefeys, M.: Finding the exact
rotation between twoimages independently of the translation. In:
European conference on computervision. pp. 696–709. Springer
(2012)
21. Larsson, V., Aström, K., Oskarsson, M.: Efficient solvers
for minimal problems bysyzygy-based reduction. In: Computer Vision
and Pattern Recognition (CVPR).vol. 1, p. 3 (2017)
22. Leonardos, S., Tron, R., Daniilidis, K.: A metric
parametrization for trifocal tensorswith non-colinear pinholes. In:
CVPR. pp. 259–267 (2015)
23. Mahmoud, A.H., El-Melegy, M.T., Farag, A.A.: Direct method
for shape recov-ery from polarization and shading. In: Image
Processing (ICIP), 2012 19th IEEEInternational Conference on. pp.
1769–1772. IEEE (2012)
24. Martyushev, E.: On some properties of calibrated trifocal
tensors. Journal of Math-ematical Imaging and Vision 58(2), 321–332
(2017)
25. Miyazaki, D., Ikeuchi, K.: Shape estimation of transparent
objects by using inversepolarization ray tracing. IEEE Transactions
on Pattern Analysis and Machine In-telligence 29(11) (2007)
26. Miyazaki, D., Kagesawa, M., Ikeuchi, K.: Transparent surface
modeling from apair of polarization images. IEEE Transactions on
Pattern Analysis and MachineIntelligence 26(1), 73–82 (2004)
27. Miyazaki, D., Shigetomi, T., Baba, M., Furukawa, R., Hiura,
S., Asada, N.: Surfacenormal estimation of black specular objects
from multiview polarization images.Optical Engineering 56(4),
041303 (2016)
28. Miyazaki, D., Tan, R.T., Hara, K., Ikeuchi, K.:
Polarization-based inverse renderingfrom a single view. In: null.
p. 982. IEEE (2003)
29. Morel, O., Stolz, C., Meriaudeau, F., Gorria, P.: Active
lighting applied to three-dimensional reconstruction of specular
metallic surfaces by polarization imaging.Applied optics 45(17),
4062–4068 (2006)
30. Murez, Z., Treibitz, T., Ramamoorthi, R., Kriegman, D.:
Photometric stereo ina scattering medium. In: Proceedings of the
IEEE International Conference onComputer Vision. pp. 3415–3423
(2015)
31. Nayar, S.K., Fang, X.S., Boult, T.: Separation of reflection
components using colorand polarization. International Journal of
Computer Vision 21(3), 163–186 (1997)
32. Ngo Thanh, T., Nagahara, H., Taniguchi, R.i.: Shape and
light directions fromshading and polarization. In: Proceedings of
the IEEE Conference on ComputerVision and Pattern Recognition. pp.
2310–2318 (2015)
33. Ponce, J., Hebert, M.: Trinocular geometry revisited. In:
Proceedings of the IEEEConference on Computer Vision and Pattern
Recognition. pp. 17–24 (2014)
34. Quan, L.: Invariants of six points and projective
reconstruction from three uncal-ibrated images. IEEE Transactions
on Pattern Analysis and Machine Intelligence17(1), 34–46 (1995)
35. Rahmann, S., Canterakis, N.: Reconstruction of specular
surfaces using polariza-tion imaging. In: Computer Vision and
Pattern Recognition, 2001. CVPR 2001.Proceedings of the 2001 IEEE
Computer Society Conference on. vol. 1, pp. I–I.IEEE (2001)
36. Shashua, A.: Algebraic functions for recognition. IEEE
Transactions on PatternAnalysis and Machine Intelligence 17(8),
779–789 (1995)
37. Smith, W.A., Ramamoorthi, R., Tozza, S.: Linear depth
estimation from an un-calibrated, monocular polarisation image. In:
European Conference on ComputerVision. pp. 109–125. Springer
(2016)
38. Taamazyan, V., Kadambi, A., Raskar, R.: Shape from mixed
polarization. arXivpreprint arXiv:1605.02066 (2016)
-
Polarimetric Three-View Geometry 17
39. Torr, P.H., Zisserman, A.: Robust parameterization and
computation of the trifocaltensor. Image and vision Computing
15(8), 591–605 (1997)
40. Tozza, S., Smith, W.A., Zhu, D., Ramamoorthi, R., Hancock,
E.R.: Linear differ-ential constraints for photo-polarimetric
height estimation. In: 2017 IEEE Inter-national Conference on
Computer Vision (ICCV). IEEE Computer Society Press(2017)
41. Walter, B., Marschner, S.R., Li, H., Torrance, K.E.:
Microfacet models for refrac-tion through rough surfaces. In:
Proceedings of the 18th Eurographics conferenceon Rendering
Techniques. pp. 195–206. Eurographics Association (2007)
42. Weng, J., Huang, T.S., Ahuja, N.: Motion and structure from
line correspondences;closed-form solution, uniqueness, and
optimization. IEEE Transactions on PatternAnalysis & Machine
Intelligence (3), 318–336 (1992)
43. Wolff, L.B., Boult, T.E.: Constraining object features using
a polarization re-flectance model. IEEE Transactions on Pattern
Analysis and Machine Intelligence13(7), 635–657 (1991)
44. Yang, L., Tan, F., Li, A., Cui, Z., Furukawa, Y., Tan, P.:
Polarimetric dense monoc-ular slam pp. 3857–3866 (2018)
45. Zhang, Z.: A flexible new technique for camera calibration.
IEEE Transactions onpattern analysis and machine intelligence
22(11), 1330–1334 (2000)