Pose Estimation using 3D View-Based Eigenspacesgroups.csail.mit.edu/vision/vip/papers/Morency_amfg03.pdfPose Estimation using 3D View-Based Eigenspaces Louis-Philippe Morency Patrik

Pose Estimation using 3D View-Based Eigenspaces

Louis-Philippe Morency Patrik Sundberg Trevor Darrell

MIT Artificial Intelligence LaboratoryCambridge, MA 02139

Abstract

In this paper we present a method for estimating the abso-lute pose of a rigid object based on intensity and depth view-based eigenspaces, built across multiple views of exampleobjects of the same class. Given an initial frame of an ob-ject with unknown pose, we reconstruct a prior model for allviews represented in the eigenspaces. For each new frame,we compute the pose-changes between every view of thereconstructed prior model and the new frame. The result-ing pose-changes are then combined and used in a Kalmanfilter update. This approach for pose estimation is user-independent and the prior model can be initialized automat-ically from any view point of the view-based eigenspaces.To track more robustly over time, we present an extensionof this pose estimation technique where we integrate ourprior model approach with an adaptive differential tracker.We demonstrate the accuracy of our approach on face posetracking using stereo cameras.

1. IntroductionEstimating the pose of a rigid object accurately and robustlyfor a wide range of motion is a classic problem in computervision and has many useful applications. We are particu-larly interested in head pose tracking and its application inview-invariant face recognition, head gesture understand-ing, and conversational turn-taking cues.

In this paper we propose a method for estimating the ab-solute pose of an object from a known class, using intensityand depth view-based eigenspaces. Our approach consistsof two steps: first we compute a prior model of the objectgiven one initial frame and then this prior model is usedto compute the absolute pose of each new frame. Here wefocus our attention on human faces, although the methodsare general enough to extend to many different classes ofrigid objects. We built our depth and intensity view-basedeigenspaces using Principal Component Analysis (PCA) for28 different viewpoints surrounding the face of 14 people.

When presented with an intensity or depth image of asubject in an unknown pose, the system first finds the viewwith minimal reconstruction error, and then uses the cor-

responding PCA coefficients to reconstruct the image atall views. This is equivalent to finding the point on themulti-view depth and intensity manifold that most closelyapproximates the observed image at some view. The recon-structed 3D multi-view model is then used as a prior modelfor absolute-pose estimation. Rigid pose tracking is easiestwhen a 3-D shape and appearance model of the object isavailable.

Given the reconstructed prior model and a new frameshowing the same subject, we estimate the new pose with atwo step process. We first compute the relative pose be-tween the new frame and each view in the prior modelusing an iterative view registration algorithm [13]. Thiscomputation uses intensity information as well as depth(if available), and amounts to estimating pose-change mea-surements between the new frame and every view in theprior model. As a final step, the pose measurements are in-tegrated using a Kalman filter to produce a final estimate forthe absolute pose [14]. This tracking framework efficientlycomputes the 6-DOF pose of the subject’s head, and couldbe provided with 2D or stereo images as input, dependingon the view registration algorithm used to do the relativepose computations.

As an extension of our approach, we integrated the re-constructed prior model in our existing Adaptive View-Based Appearance Model (AVAM) tracking framework[14]. This method creates a user-specific view-based modelonline during tracking. It can estimate the pose of an objectaccurately and with bounded drift, relative to the first frame.By integrating the prior model in the AVAM framework, weget a robust tracker able to initialize automatically and trackobject outside the pose space defined in our prior model.

Section 2 reviews previous work and how it relates tothis paper, and Section 3 describes how we construct theview-based eigenspaces. We then in Section 4 present thealgorithm to create a prior model of the person of interestgiven the initial image of an image sequence. Section 5presents our technique for 6-DOF pose estimation using theview registration and Kalman filter framework. Section 5.1describes the integrated framework with AVAM. Finally, inSection 6, we show results for a head tracking task withdepth and intensity input from a commercial stereo camera,

and compare the accuracy of our pose estimation techniquewith that of another technique[18].

2. Previous Work

Pose estimation is possible from a single 2-D view–e.g,using color and coarse template matching [2, 16], patternclassifiers [15], or using graph matching techniques [11]–but techniques which can exploit a 3-D model are generallymore accurate. 3-D representations model the appearanceof objects more closely, and thus can lock on to a subjectmore tightly. Textured geometric 3D models [10, 1] havebeen used for tracking; because the prior 3D shape modelsfor these systems do not adapt to the user, they tend to havelimited tracking range.

Deformable 3D models fix this problem by adapting theshape of the model to the subject [9, 12, 4, 6]. These ap-proaches maintain the 3D structure of the subject in a statevector which is updated as images are observed. These up-dates require that correspondences between features in themodel and features in the image be known. Reliably com-puting these correspondences is difficult, and the complex-ity of the update grows quadratically with the number of3D features, making the updates expensive [9]. Brand de-veloped a 3-D morphable model which is able to track fea-tures while simultaneously estimating the underlying shapemodel [3]. In general, existing approaches to 3-D modelestimation for tracking presume a single-viewpoint modelof image appearance, which will not be valid for non-lambertian objects.

A view-based approach to 3-D modeling and trackinghas several advantages over mesh or volumetric shape mod-els. The relative pose of constituent range observations canbe adjusted dynamically during model formation. It caneasily represent varying levels of detail on an object, and itdirectly captures non-lambertian appearance on the surfaceof an object [14].

Below, we describe a pose estimation algorithm usingview-based eigenspaces with depth and intensity compo-nents. View-based models for object recognition usingeigenspaces were described in [17], which constructed aseparate PCA model for sets of images at given views. [5]developed a multi-view active appearance model that de-scribed shape and texture variation across views; object ap-pearance was matched using the closest view and pose in-ferred with a linear projection of model coefficients.

Recently the reconstruction distance to a set ofeigenspaces at different views was used to interpolate headpose; the relationship between a set of approximate correla-tion scores and object pose can be learned from training ex-amples [18]. Our approach differs from this work in that welearn a joint eigenspace across views, intensity, and depthimages, and that we use the model only to reconstruct a

multi-view model close to the observed object. We do notuse the correlation scores or reconstruction error from eachview to infer pose; instead we compute the pose-changesbetween every view of the reconstructed prior model andthe new frame. The resulting pose-changes are then com-bined and used in a Kalman filter update.

The Adaptive View-based Appearence Model describedin [14] is a relative-pose tracker which combines differen-tial tracking with keyframe-based tracking. Adaptive view-based models can be acquired online during the tracking.The uncertainty in the pose of a keyframe shrinks overtimeas the keyframe is revisited. Hence, the uncertainty in thepose estimate of new frame registered against a keyframe isbounded. The resulting tracker has bounded drift and canbe used to track heads undergoing large motion for a longtime. By creating the adaptive appearance model online,the tracker gives accurate relative pose but doesn’t have anymechanism to estimate absolute pose [14].

3. 3D View-based EigenspacesWe wish to learn a multi-view depth and intensity modelwhich is user-independent and can be used to initialize posetracking . We also want a view-based model that can recon-struct a multi-view manifold of the object given only oneview. Ideally, we could recreate the depth manifold givenonly an intensity image as input.

To achieve these goals, we define our view-basedeigenspaces modelP as:

P = I ,VI , Z,VZ

whereI andZ are the mean intensity and depth for all theviews; VI andVZ are the intensity and depth eigenspacesof our model. To navigate in our model, we define windowsPi in the eigenvector matrices for each viewi of our model:

Pi = Ii, VIi, Zi, VZi

, εi

where Ii and Zi are the mean intensity and depth imagesfor this view,εi is the pose of that view andVIi

andVZiare

windows in the eigenspace matrices. Note thatVIi andVZi

are not eigenspaces since we defined our eigenspacesVI

andVZ over all the views. In our case, poses of rigid bodyare represented asε = [ T x T y T z Ωx Ωy Ωz ],a 6 dimensional vector consisting of the translation and theinstantaneous rotation.

3.1. Eigenspaces AcquisitionWe want to generate a user-independent view-based modelthat can render every view of the object given a correctmatch with one of the views. The view-based eigenspacesP can be learned from multiple adaptive view-based appear-ance modelsM1,M2, ...,Mn. Following the definition

stated in [14], an adaptive view-based appearance model isdefined as

M = Ii, Zi, εi,Λ

whereIi, Zi are the intensity and depth images at eachview i, εi is the pose of each key frame modelled with aGaussian distribution, andΛ is the covariance matrix overall random variablesεi.

For each modelMj , we concatenate the intensity anddepth images of all views in two vectorsIj andZj :

Ij =[

Ij1 Ij

2 · · · Ijm

]Zj =

[Zj

1 Zj2 · · · Zj

m

]where Ij

i and Zji are segmented intensity and depth im-

ages from the appearance modelMj at poseεi, andm isthe number of views. All the segmented images have thesame size and are stored in one-dimensional vectors. Wecan compute the average vectors:

I = 1n

∑nj=1 Ij Z = 1

n

∑nj=1 Zj (1)

and then stack all the normalized intensity and depth vectorsinto two matrices:

I =[ (

I1 − I) (

I2 − I)

· · ·]T

Z =[ (

Z1 − Z) (

Z2 − I)

· · ·]T

Since we want to be able to reconstruct the intensity anddepth images from only one intensity image, we must usethe same set of weights for the intensity and the depth eigen-vectors. To achieve that, we apply SVD decomposition onI = UIDIVT

I and compute the corresponding depth eigen-vectors by applying the same weights:

VTZ = D−1

I U−1I Z

This approach allows us to create a prior model with bothintensity and depth even when no stereo information isavailable. Although the resulting depth basis vectors ofVZare not optimal, there is a strong correlation between the in-tensity and depth images that provides justification for thisapproach.

Figure 1 shows the mean face of our view-basedeigenspaces built using adaptive view-based models of 14people. Each adaptive model contains 28 views of one per-son: 7 views along the X axis by 4 views along the Y axis.All adjacent views are separated by 10. By looking closelyat the depth images, we can see that the chin is closer whenlooking at 20 up. When looking on the side, we can see asmall bump representing the nose. Such subtle details canbe important during tracking.

Figure 2 shows the first three intensity eigenvectors dis-played for the 7 horizontal views. We can see in the secondeigenvector the variations for the nose and the eye shadow.The third eigenvector presents some lip variation.

-30˚ -20˚ -10˚ 0˚ 10˚ 20˚ 30˚

20˚

10˚

0˚

-10˚

-10˚

0˚

10˚

20˚

Figure 1:Top: 28 views of the average intensity manifold. Bot-tom: 28 views of the average depth manifold (white means closer).

Figure 2: The first three intensity eigenvectors (rows) partiallydisplayed for the 7 horizontal views (colomns).

4. Prior Model Reconstruction

The purpose of prior model reconstruction is to generate aset of views for use in the pose estimation module. Given asingle example frame near one of the views in our modelP,we want to reconstruct all the other available views of ourmodel. In our case, given one image, we can recreate the 27other views including the depth images.

The new unsegmented frameIt, Zt is preprocessed tofind a region of interest for the object. This can be doneusing motion detection, background subtraction, flesh colordetection or a simple face detector [19].

For each viewi of our model and for each subregionI ′t, Z ′t of the same size as the views inP inside the re-gion of interest, we find the vector~wi that minimizes

Ei = |I ′t − Ii − ~wi · VIi|2, (2)

The minimization is straightforward using linear leastsquares.

From the eigenvector weights~wi, we first reconstruct theintensity and depth images

IRi= Ii + ~wi · VIi

(3)

ZRi= Zi + ~wi · VZi

(4)

The reprojection step is done for every viewi. Afterthe reconstruction of all intensity and depth imagesIRi

andZRi

, we search for the best projection minimizing the cor-relation function:

(I ′t − I ′t) · (IRi− IRi

)|I ′t − I ′t||IRi − IRi |

+ λ(Z ′t − Z ′t) · (ZRi

− ZRi)

|Z ′t − Z ′t||ZRi − ZRi |

whereλ is constant to compensate for the difference be-tween intensity measurements(brightness levels) and thedepth measurements(mm). If the depth image is not avail-able thenλ is set to 0.

From the correlation function, we get a correlation scoreci for each view and each subregionI ′t, Z ′t. The lowestcorrelationc∗i over all views and all subregions correspondsto the best match. Using the weights~w∗ of the best match,we again reconstruct the intensity and depth images of theobject in all the views using eq. (3) and (4). The outputof the matching algorithm is the set of reconstructed framesI∗i , Z∗i , the poseε∗i of the best view-based eigenspace andthe associated correlation scorec∗i . We can define a priorview-based appearance model

MP = IPi , ZPi , εPi,ΛP

where the images and poses are copied directly from re-constructed frames and associated poses, and the covariancematrixΛP is initialized as the identity matrix times a smallconstant.

Figure 3 shows reconstructions for 2 different peo-ple. Each reconstruction was done using view-basedeigenspaces that exclude the person reconstructed. The tophalf shows a reconstruction where the example image is ori-ented near the view 0 around X axis and 0 around the Yaxis. The reconstructed views displayed in the figure are thehorizontal view. The second reconstruction uses an exam-ple image at 20 around X axis and 20 around the Y axis.The reconstructed views are at -10 around the Y axis.

5. 6-DOF Absolute Pose EstimationIn this section we present our technique to estimate theabsolute pose of a rigid object using 3D view-basedeigenspaces as a prior model. Figure 4 presents an overviewof our pose estimation algorithm. Our approach is separatedin two steps: first we compute a prior model of the subject

Figure 3:Model reconstruction from a frontal view (top half) anda rotated view (bottom half). Both reconstructions (bottom rows)are compared with ground truth (top rows).

given one initial frame and then this prior model is usedwith each new frame to compute the absolute pose.

During the initialization stage, each new frameIt, Ztis projected into the view-based eigenspace model as de-scribed in section 4. If the best correlation scorec∗i is largerthan a thresholdk, the prior model is created.

When depth information is available in the new frame,we can register the frames using a hybrid error functionwhich combines robustness of the ICP (Iterative ClosestPoint) algorithm and the precision of the normal flow con-straint (NFC) [13]. When only 2D images are availablean iterative approach like [6] may be used to give an ap-propriate set of pose-change measurement. We model apose-change measurementδt

s as having come fromδts =

εt − εs + ω whereεs is the pose estimate associated withthe views of our prior model andω is Gaussian.

To estimate the poseεt of the new frame based on thepose-change measurements, we use the Kalman filter for-mulation described in [14] where the state vectorX is pop-ulated with the pose variablesεt, εP1 , εP2 , . . . and theobservation vectorY is populated with the pose-changemeasurementsδt

P1, δt

P2, . . .. The covariance between the

components ofX is denoted byΛX .The Kalman filter update computes a prior for

p(Xt|Y1..t−1) by propagatingp(Xt−1|Y1..t−1) one step for-ward using a dynamic model. Each pose-change measure-mentyt

s ∈ Y between the current frame and a base frame ofX is modelled as having come from:

yts = CX + ω,

frame 0

ε0

ε1

ε2

ε3

ε4

ε5

ε6

new frame

δ0

δ1

δ2

δ3

δ4

δ5

δ6

view-basedeigenspace

prior model

view registration

modelcreation

view registration

view registration

view registration

view registration

view registration

view registration

Figure 4: Overview of the prior model and its usage. Section 3describes how the view-based eigenspaces are created, Section 4describes the model creation step that is done once at the beginningof each sequence, and section 5 describes view registration.

C =[

I 0 · · · −I · · · 0],

whereω is Gaussian. Each pose-change measurementyts

is used to update all poses using the Kalman Filter stateupdate:

[ΛXt]−1 =

[ΛXt−1

]−1 + C>Λ−1yt

sC (5)

Xt = ΛXt

([ΛXt−1

]−1Xt−1 + C>Λ−1yt

syt

s

)(6)

We define our observations variablesδts as a pose-change

measurement between the new frame and a base frame inX .

5.1. Integration with AVAMIn this section we present an extension of the 6-DOF ab-solute pose estimator where we integrate the reconstructedprior model inside an Adaptive View-based AppearanceModel (AVAM) tracking framwork. In the AVAM frame-work, user-specific keyframes are added in the model dur-ing the tracking. One of the main advantages of the AVAMframework is that pose estimation of the new frameIt, Ztand pose adjustments of the view-based modelM are per-formed simultaneously. The original AVAM described in[14] is a relative-pose tracker which combines differen-tial tracking with keyframe-based tracking. By integrat-ing the prior model with the AVAM framework, we obtainan absolute-pose tracker with accurate pose estimates andbounded-drift.

-10

-8

-6

-4

-2

0

2

4

6

8

10

12

1 8

15

22

29

36

43

50

57

64

71

78

85

92

99

10

6

11

3

12

0

12

7

13

4

14

1

14

8

15

5

16

2

16

9

17

6

18

3

19

0

19

7

20

4

21

1

21

8

22

5

23

2

23

9

Ro

tati

on

aro

un

d X

ax

is (

de

gre

e)

Inertia Cube2

Our prior model

OSU

-40

-30

-20

-10

0

10

20

30

1 8

15

22

29

36

43

50

57

64

71

78

85

92

99

10

6

11

3

12

0

12

7

13

4

14

1

14

8

15

5

16

2

16

9

17

6

18

3

19

0

19

7

20

4

21

1

21

8

22

5

23

2

23

9

Ro

tati

on

aro

un

d Y

ax

is (

de

gre

e)

Inertia Cube2

Our prior model

OSU

-20

-15

-10

-5

0

5

10

15

20

1 8

15

22

29

36

43

50

57

64

71

78

85

92

99

10

6

11

3

12

0

12

7

13

4

14

1

14

8

15

5

16

2

16

9

17

6

18

3

19

0

19

7

20

4

21

1

21

8

22

5

23

2

23

9

Ro

tati

on

aro

un

d Z

ax

is (

de

gre

e)

Inertia Cube2

Our prior model

Figure 5: Comparison of the pose estimation results of the bestreconstruction (4 eigenvectors) with the InertiaCube2 and with ourreimplementation of the OSU pose estimator. The ground truthestimates from the Inertial Cube2 are shown with 3 error bars.

Since the AVAM framework also uses a Kalman filter toupdate the poses, we can extend the Kalman filter state vec-tor to include the key frame pose variables and the previousframe pose variable:

X =[

εt εt−1 εM1 εM2 . . . εP1 εP2 . . .]>

By integrating the prior model in the AVAM framework,we get a robust tracker able to initialize automatically andestimate absolute pose of the object.

6. ExperimentsWe designed our experiments to demonstrate the accuracyof our approach on estimating the relative and absolute poseof a user’s head in 3D. All the experiments were done usinga Videre Design stereo camera [7].

We constructed the view-based eigenspaces using 14view-based appearance models acquired with the originaltracker described in [14]. Each participant was aligned fac-ing the camera, and then asked to rotate his head along theX and Y axes. We configured the tracker to tesselate therotation space at 10 degree intervals. In this fashion, a setof 28 intensity and depth image pairs was created for eachparticipant. We then manually cropped the faces to 32x32pixels, while keeping alignment across participants. Figure1 shows the average face in all 28 orientations, and Figure2 shows 3 of the 13 eigenvectors that we use to create priormodels.

To analyze our algorithm quantitatively, we comparedthe pose estimates of our system with those of an InterSenseInertiaCube2 sensor [8]. The InertiaCube2 is an inertial 3-DOF orientation tracking system, which we mounted on theinside of a construction hat that was worn by a test subjectduring tracking. The sensor works by measuring the direc-tion of gravity and the Earth’s magnetic field, and is driftlessalong the X and Z axes. However, the Y axis (pointing up)can suffer from errors due to drifting. InterSense reports adynamic accuracy of 3RMS.

6.1. 6-DOF Absolute Pose DetectionTo analyze the accuracy of our view-based eigenspacemodel for pose estimation, we recorded 2 sequences withground truth poses using an InertiaCube2 sensor. Sequence1 contains 245 frames and sequence 2 contains 155 frames.Because the purpose of the experiment was to evaluate theaccuracy of the pose estimation algorithms, poses in bothsequences are constrained to the rotation space of the priormodel. The pose estimation algorithm described in section5.1 is applied independently for each frame.

For comparison, we also reimplemented a 2-dimensionalversion of the OSU system [18]. Our implementation ofthe OSU estimator used 7 different eigenspaces in the hori-zontal direction, at poses between -30 and 30 degrees withequal spacing, and 4 different eigenspaces in the vertical di-rection, between -20 and 10 degrees. This is the same rangespanned by the eigenspaces we used in the other part of thepaper. However, using this method pose estimation is doneindependently for the two degrees of freedom.

In the OSU pose estimation framework, the first step isto find the face in the image. This is accomplished by com-puting correlation scores with the mean face for each of theorientations in the model for every possible location in alarge region of interest around the face. We then normalizethe detected face to have zero mean and unit variance, andproject it onto each eigenspace. The eigenspace that canrepresent the largest fraction of the energy of the input facedetermines a first coarse estimate of the pose. Finally, anincremental estimate of the pose is computed. The incre-mental estimates for the horizontal and vertical directionsare given by

∆θh =rh,23 − 0.989

0.073(7)

∆θv =rv,23 − 0.945

0.066, (8)

whererh,23 andrv,23 are the ratios of the energy capturedby the second and third best candidates for the coarse poseestimation in the horizontal and vertical directions, respec-tively. The numerical constants were computed using linearleast squares from a training sequence tagged with groundtruth. The incremental estimate is always taken to be in

3

3.5

4

4.5

5

5.5

6

0 2 4 6 8 10 12 14

Number of eigenvector used for model reconstruction

Ma

gn

itu

de

of

RM

S e

rro

r fo

r a

ll 3

ro

tati

on

s (

de

gre

es

)

Figure 6: Variation of the RMS error of pose estimation on se-quence 1 as we change the number of eigenvectors used to createthe prior model.

Our prior model X Y ZSequence 1 1.62 2.55 1.67

Sequence 2 1.04 3.91 1.44

OSU X Y ZSequence 1 2.30 4.46

Sequence 2 1.74 3.01

Table 1: RMS error in degrees for pose estimation comparingour prior model pose estimation (top) and the OSU pose estima-tion(bottom). The OSU pose estimator doesn’t return any estimatefor the rotation around the Z axis.

the direction of the second best coarse pose estimate, and iscapped at 5.0 degrees.

Figure 5 shows the pose variations recorded from theground truth sensor compared with the OSU pose estima-tor and our prior model pose estimator. Figure 6 shows themagnitude of the RMS error for the 3 rotations when vary-ing the number of eigenvectors used for the reconstructionof the prior model. We found that, in our case, 4 eigen-vectors was a good trade-off between model expressivenessand model over-fitting.

Table 1 presents a comparison of the RMS error for thepose estimation using our prior model and the OSU poseestimator. The InertiaCube2 is not drift-free around the Yaxis. The average RMS error of the prior model pose esti-mator for all 3 axis is 3.88 which is close to the accuracyof the InertiaCube sensor (3). The OSU pose estimatordoesn’t model rotations around the Z axis hence gives noestimates. If we fix the OSU Z-axis output to 0, the RMSerror around the Z axis is approximately 3.9 for both se-quences. Note that our prior model pose estimator is able tohandle rotation around the Z axis even though the trainingdata for our prior model did not include frames with rotationaround the Z axis.

6.2. Integrated AVAM & Prior ModelOne of the main advantages of the user-independent view-based eigenspace model when inserted in our Kalman fil-ter update is that we can estimate absolute poses. This isa considerable advantage compared to tracking techniquesare initialized manually with ad-hoc techniques.

To demonstrate the performance of the pose estimator,we recorded a sequence of approximately 2 mins at 7Hz fora total of 800 frames. The user moved freely from left toright, front to back, rotated his head left and right, top tobottom, and also tilted his head. The purpose of this se-quence is to show how our integrated technique can give anaccurate estimate of the absolute pose in an unconstrainedenvironment. In the video sequence, which can be foundat http://www.ai.mit.edu/projects/watson/, we represent theestimate of the absolute pose by a cube around the headof the user. The thickness of the cube is inversely propor-tional to the variance of the absolute pose estimate. Thered squares below the cube represent the number of baseframes used to compute the estimate. This video shows theresults of our approach which integrates differential track-ing, adaptive view-based appearance model and view-basedeigenspace prior model.

Table 2 presents the tracking results for different config-urations of the tracker. The first row represents results usingonly the differential tracker. Differential tracking drifts af-ter a certain amount of time, leading to high RMS error.The second row represents the pose estimation algorithmdescribed in Section 5. Since the movements in this se-quence were not constrained to the rotation space of ourprior model, the performance of this approach is poor. Thethird row shows the performance of the original differen-tial tracker and adaptive model. This tracking technique hasshown to yield good results when estimating relative posebut without a good prior model this technique does poorly interms of absolute pose estimation. The fourth row presentsthe results of the differential tracker with the prior modelpose estimator. This gives better results then the prior modelalone since the differential tracker can give a good pose es-timate even outside the rotation space of the prior model.Also, the differential tracker acts as a dynamic model in theKalman filter which helps to smooth the estimates of theprior model pose estimator. Finally, the last row presentsthe results of the integrated pose estimation and tracking.In that configuration, the prior model estimates the absolutepose during the initialization period which helps the accu-racy of the online adaptive appearance model.

7. Summary and ConclusionsWe described a new technique for pose estimation based onview-based models. View-based models capture a rich rep-resentation of object shape and surface appearance across a

Technique X Y Z TotalDiff 10.80 21.96 14.74 28.56

Prior 5.12 10.26 3.47 11.98

Diff+Adapt 6.48 8.15 2.23 10.65

Diff+Prior 2.64 5.39 2.43 6.48

Diff+Adapt+Prior 2.46 5.00 2.46 6.09

Table 2:RMS error for each tracking technique.

wide range of pose change. We learned a multi-view, depthand intensity eigenspace model which provides a set of priorkeyframes for pose estimation and tracking customized to anew user. A Kalman filter framework combines pose es-timates using both prior and adaptive keyframes accordingto estimates of uncertainty for each. The use of our poseestimation technique greatly reduces the absolute error inview-based tracking, which was previously limited by thecoarse pose estimate of generic face detectors used for ini-tialization. We demonstrate the accuracy of our integratedapproach on face pose tracking using stereo cameras.

References[1] S. Basu, I.A. Essa, and A.P. Pentland. Motion regulariza-

tion for model-based head tracking. InProceedings. Inter-national Conference on Pattern Recognition, 1996.

[2] S. Birchfield. Elliptical head tracking using intensity gradi-ents and color histograms. InIEEE Conference on ComputerVision and Pattern Recognition, pages 232–237, 1998.

[3] M. Brand and R. Bhotika. Flexible flow for 3d nonrigidtracking and shape recovery. InProc. Conf. on ComputerVision and Pattern Recognition, volume 1, pages 315–322,2001.

[4] A. Chiuso, P. Favaro, H. Jin, and S. Soatto. Structure frommotion causally integrated over time.IEEE Trans. on PatternAnalysis and Machine Intelligence, 24(4):523–535, April2002.

[5] T.F. Cootes, G.J. Edwards, and C.J. Taylor. Active appear-ance models.IEEE Trans. on Pattern Analysis and MachineIntelligence, 23(6):681–684, June 2001.

[6] D. DeCarlo and D. Metaxas. Adjusting shape parametersusing model-based optical flow residuals.IEEE Trans. onPattern Analysis and Machine Intelligence, 24(6):814–823,June 2002.

[7] Videre Design. MEGA-D Megapixel Digital Stereo Head.http://www.ai.sri.com/ konolige/svs/, 2000.

[8] InterSense Inc.InertiaCube2. http://www.intersense.com.

[9] T. Jebara and A. Pentland. Parametrized structure from mo-tion for 3D adaptive feedback tracking of faces. InIEEEConference on Computer Vision and Pattern Recognition,1997.

[10] M. LaCascia, S. Sclaroff, and V. Athitsos. Fast, reliable headtracking under varying illumination: An approach based on

registration of textured-mapped 3D models.IEEE Trans. onPattern Analysis and Machine Intelligence, 22(4):322–336,April 2000.

[11] M. Lades, J.C. Vorbruggen, J. Buhmann, J. Lange, C. von derMalsburg, R.P. Wurtz, and W. Konen. Distortion invariantobject recognition in the dynamic link architecture.IEEETransactions on Computers, 42(3):300–311, March 1993.

[12] P. F. McLauchlan. A batch/recursive algorithm for 3D scenereconstruction.Conf. Computer Vision and Pattern Recogni-tion, 2:738–743, 2000.

[13] L.-P. Morency and T. Darrell. Stereo tracking using icp andnormal flow. InProceedings International Conference onPattern Recognition, 2002.

[14] L.-P. Morency, A. Rahimi, and T. Darrell. Adaptive view-based appearance model. InProceedings IEEE Conf. onComputer Vision and Pattern Recognition, 2003.

[15] J. Ng and S. Gong. Composite support vector machines fordetection of faces across views and pose estimation.Imageand Vision Computing, 20:359–368, 2002.

[16] N. Oliver, A. Pentland, and F. Berard. Lafter: Lips and facereal time tracker. InComputer Vision and Patt. Recog., 1997.

[17] A. Pentland, B. Moghaddam, and T. Starner. View-based andmodular eigenspaces for face recognition. InProc. of IEEEConf. on Computer Vision and Pattern Recognition, Seattle,WA, June 1994.

[18] S. Srinivasan and K.L. Boyer. Head pose estimation usingview based eigenspaces. InProceedings. 16th InternationalConference on Pattern Recognition, pages 302–305, 2002.

[19] P. Viola and M. Jones. Robust real-time face detection. InProceedings International Conference on Computer Vision,page II: 747, 2001.