Top Banner
Towards Practical Visual Servoing in Robotics Romeo Tatsambon Fomena Camilo Perez Quintero Mona Gridseth Computer Vision and Robotics Research Group Department of Computing Science University of Alberta Edmonton, AB, Canada {tatsambo, caperez, gridseth, jag}@ualberta.ca Martin Jagersand Abstract—Visual servoing was introduced in robotics nearly 4 decades ago. However until now, there are still only a handful of known examples of application of this technique in addressing real word robotics problems such as disaster response, assistance for elderly or handicapped people, etc. As the world is moving towards the use of robotics to improve quality of life, it is time to assess the challenges involved in applying visual servoing to solve real world problems. This paper presents an overview of these challenges, by asking the question what are the missing components for practical visual servoing? and by providing practical possible solutions for these components. Illustration of these challenges and our current practical solutions are given using our 7-DoFs Barrett WAM Arm. Keywords-visual servoing; visual tracking; visual task speci- fication; human-robot interface; I. I NTRODUCTION Despite interesting advances over the past 5 decades in research and development, robotic manipulation has yet to become generally useful in human environments. Let us consider a simple breakfast preparation task to be realized by a personal assistant robot. Figure 1 illustrates this task. This figure mainly illustrates the difficulties of the current existing visual tracking algorithms to provide reliable mea- surements of the position of the cereal box. For an average human these kinds of task are naturally achievable. However, for a robot several key challenges have to be addressed: visual tracking, task sequencing, path planning and task precision. In fact, robotics is not yet capable of solving real world problems. This is because industrial robot technology has not been easy to adapt outside engineered workcells. This lack of adaptation can be explained by the fact that: Our everyday natural environment is complex. This makes it harder to design an adequate representation usable for control and navigation of robots Most of robotics research has focused more on mecha- tronics designs and on theoretical developments than on practical software developments. This results in an increasing gap between the advertized robotics and its real capabilities in practice. (a) (b) Figure 1. Pouring cereal for breakfast using a mobile robotic platform, here the Barrett WAM mounted on a Segway base: (a) Tracking and (b) different steps involved to achieve the task. Triggered by recent world disasters, especially the Fukushima nuclear accident, the current DARPA robotics
8

Towards Practical Visual Servoing in Roboticstatsambo/publications/TowardsPracticalVisualS… · Towards Practical Visual Servoing in Robotics Romeo Tatsambon Fomena Camilo Perez

Oct 18, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Towards Practical Visual Servoing in Roboticstatsambo/publications/TowardsPracticalVisualS… · Towards Practical Visual Servoing in Robotics Romeo Tatsambon Fomena Camilo Perez

Towards Practical Visual Servoing in Robotics

Romeo Tatsambon Fomena Camilo Perez Quintero Mona Gridseth

Computer Vision and Robotics Research GroupDepartment of Computing Science

University of AlbertaEdmonton, AB, Canada

{tatsambo, caperez, gridseth, jag}@ualberta.ca

Martin Jagersand

Abstract—Visual servoing was introduced in robotics nearly4 decades ago. However until now, there are still only ahandful of known examples of application of this techniquein addressing real word robotics problems such as disasterresponse, assistance for elderly or handicapped people, etc. Asthe world is moving towards the use of robotics to improvequality of life, it is time to assess the challenges involved inapplying visual servoing to solve real world problems. Thispaper presents an overview of these challenges, by askingthe question what are the missing components for practicalvisual servoing? and by providing practical possible solutionsfor these components. Illustration of these challenges and ourcurrent practical solutions are given using our 7-DoFs BarrettWAM Arm.

Keywords-visual servoing; visual tracking; visual task speci-fication; human-robot interface;

I. INTRODUCTION

Despite interesting advances over the past 5 decades inresearch and development, robotic manipulation has yet tobecome generally useful in human environments. Let usconsider a simple breakfast preparation task to be realizedby a personal assistant robot. Figure 1 illustrates this task.

This figure mainly illustrates the difficulties of the currentexisting visual tracking algorithms to provide reliable mea-surements of the position of the cereal box. For an averagehuman these kinds of task are naturally achievable. However,for a robot several key challenges have to be addressed:visual tracking, task sequencing, path planning and taskprecision.

In fact, robotics is not yet capable of solving real worldproblems. This is because industrial robot technology hasnot been easy to adapt outside engineered workcells. Thislack of adaptation can be explained by the fact that:• Our everyday natural environment is complex. This

makes it harder to design an adequate representationusable for control and navigation of robots

• Most of robotics research has focused more on mecha-tronics designs and on theoretical developments thanon practical software developments. This results in anincreasing gap between the advertized robotics and itsreal capabilities in practice.

(a)

(b)

Figure 1. Pouring cereal for breakfast using a mobile robotic platform,here the Barrett WAM mounted on a Segway base: (a) Tracking and (b)different steps involved to achieve the task.

Triggered by recent world disasters, especially theFukushima nuclear accident, the current DARPA robotics

Page 2: Towards Practical Visual Servoing in Roboticstatsambo/publications/TowardsPracticalVisualS… · Towards Practical Visual Servoing in Robotics Romeo Tatsambon Fomena Camilo Perez

challenge aims at improving disaster response operations inorder to bridge the gap between the current capability ofrobotics and the high health risks faced by rescue workers.The Emergence of ROS (Robot Operating System) goesalong the way of improving the current state of the artsoftware development in robotics by aiming at reducing inte-gration costs through standardization [1]. ROS is world-classmiddleware which allows robot programmers and industrialsto quickly and easily make good solutions reusable [2].

Following this current trend in robotics, it is time toadvance visual servoing control techniques towards morepractical and usable implementations. Very few researchgroups have presented fully integrated visual servoing sys-tems applied to human needs. A noticeable example evalu-ated with upper body disabled persons is presented in [3]where a user explicity selects an object and the systemautonomously retrieves it. The challenges which need tobe addressed range from the human-robot interface to therobot control systems. While the first one is concerned witheasy to use and intuitive interfaces and their evaluations, thesecond is concerned with the motion control of the robot. Inaddition to this, issues such as visual task specification andverifiability, and visual tracking need to be solved. Althoughsome solutions for the tracking problem exist, there is stillnot yet a way to design reliable trackers for some scenes andto deal with tracking failure occuring during visual servoing.The task specification problem raises the question of how torepresent a robot workspace task in the camera image space?

This paper reviews visual servoing techniques andpresents some practical solutions to the above mentionedproblems. Going in the opposite order from how a humanoperator encounters a system, we first present the core ofthe system before looking at the human-robot interface.

This paper is organized as follows. Section II recalls visualservoing in robotics. In this section, we present both thecalibrated and the uncalibrated visual servoing approaches.Section III reviews some tracking techniques that could beapplied to visual servoing. Section IV presents an intuitiverobot command interface where visual evidence of conver-gence can be proved to be equivalent to convergence in therobot workspace. Section V presents touchscreen and gestureinterfaces as a way to improve the intuiveness of the visualservoing command interface. Finally, this section describesexperimental validations of these intuitive interfaces usingour 7 DoFs Barrett WAM Arm in the context of a task wherekitchen utensils and kids toys spread over a table are sortedin 2 different bins.

II. VISUAL SERVOING IN PRACTICE

In this section we review visual servoing control tech-niques in robotics and we present the issues which limit theirapplication in practice. We also propose a possible practicalapproach to visual servoing in robotics.

(a)

(b)

Figure 2. Positioning the Barret WAM Arm with respect to a goal posedefined by the goal image of the object of interest: (a) block diagramof an image-based control and (b) closed-loop system trajectories. Theimage of the Barret WAM Arm+attached camera obtained courtesy of AzadShademan.

1) Review: Visual servoing consists of using feedbackprovided by one or several vision sensors to influence themotion of a dynamic system [4], [5], [6]. Generally, thetask in visual servoing consists of controlling the pose(position and orientation) of a rigid body relative to anobject of interest as shown by Figure 2. This figure showsthe positioning in the world space of the end-effector ofa robotic arm, using the vector of image coordinates offour points in the image space to generate control signalsin the robot motor space. To achieve a visual servoingpositioning task, a parameterization s(q, t) ∈ Rm of a set ofvisual features1, where q represents the rigid body state, isselected from the image of the object of interest. To be moreprecise, the visual measurements vector s(q, t) represents

1A set of visual features is a set of primitives in the image plane whichcorrespond to the projection of a tri-dimensional geometrical primitives(3D points, 3D lines, planes, etc) [7] and their geometric relationship(collinearity, etc).

Page 3: Towards Practical Visual Servoing in Roboticstatsambo/publications/TowardsPracticalVisualS… · Towards Practical Visual Servoing in Robotics Romeo Tatsambon Fomena Camilo Perez

a parameterization of a configuration of a set of visualfeatures. The primary control principle is to regulate theerror e(q(t), t)= s(q(t), t)− s∗(q∗(t), t), where the goalvalues s∗(q∗, t) corresponds to the goal state q∗ of thesystem. That is what we want to achieve

limt→∞

‖e(t)‖ = 0.

One approach to achieving this objective is to compute theerror dynamics differential equation

de

dt= J(t)q(t) +

∂e(t)

∂t, (1)

where q(t) is the system control inputs, J(t)= ∂e(t)∂q(t) is the

feature Jacobian matrix and ∂e(t)∂t represents the contribution

of a possible autonomous motion of the object of interest.The control law is built from expression (1) using theknowledge of J(t) and ∂e(t)

∂t which in general are notdetermined exactly.

There are three main approaches in visual servoing: 3D,hybrid and image-based visual servoings. In 3D visual servo-ing, visual features in the 3D Cartesian space, representedby SE(3)= R3 × SO(3), are used as control inputs [8].These features are obtained from the relative pose of thecamera with respect to the object of interest. This pose canbe recovered knowing the model of the object (for examplethe Euclidean distances between salient points of the object).In hybrid (or so-called 2 1/2 D) visual servoing whichcombines image and 3D data, the orientation in SO(3)(which is the 3D data) of the camera between the current andthe goal poses, are generally obtained via the estimation ofan Euclidean homography transformation (mapping betweenpoints in two Euclidean planes) between the current and goalimages [9]. This method has the advantage over 3D visualservoing to not need the model of the object. However thereis a drawback to the 2 1/2 D visual servoing method: it ismore sensitive to image noise (like 3D visual servoing) thanimage-based visual servoing, which uses directly featuresextracted in the image as control inputs. The image-basedvisual servoing approach, as opposed to 3D and hybridvisual servoings, uses direct image feedback, and is thus verysuitable for unstructured environment, for instance withoutcamera and object models [10].

2) A possible practical approach: Visual servoing haspotential applications in different areas: in space robotics, fortelemanipulation and on-orbit-servicing, in medical robotics,for ultrasound imaging diagnosis interventions, in aerialrobotics, for target tracking and position control, etc. Inorder to apply visual servoing in practice, we propose asemi-autonomous control approach where the robot and thehuman-operator collaborate together to achieve a task. Theprincipe of this approach is that the robot autonomouslydeals with its dynamics for precise positioning, whereas thehuman operator deals with task initialization and monitoring.

Because of the above mentioned advantages of image-based visual servoing, this approach is chosen as our au-tonomous control system. In this approach we can distin-guish between the calibrated and the uncalibrated methods.As oppose to the uncalibrated method, the calibrated methodrequires vision sensor (classical cameras and 3D camerassuch as Kinect) and robot-camera calibration as well as therobot model. The latter calibration, which can be obtainedthrough a complex process [11], is inherent to errors whichmay limit the precision and the accuracy of a visual servoingtask. In addition, errors in the robot model usually lead topoor precision of the positioning as it is the case in [3] for thegross motion. That is why, due to the complexity involvedin the robot-camera calibration and the modeling of high di-mension dynamic systems such as robot arms and humanoidrobots which can be used for disaster response, uncalibratedvisual servoing (UVS) seems to be the appropriate candidateapproach for practical visual servoing.

However, up to date, designing an uncalibrated image-based visual servoing controller is still considerably chal-lenging because it involves 3 main ingredients: an adequateimage tracking method, a formal learning algorithm todesign an update of the Jacobian J(t) = ∂e(t)∂q ensuringcontrol convergence and, more importantly an image-basedtask specification and verifiability approach. Other issuessuch as path planning appears when considering navigationtasks. In the case of manipulation of objects on the top of atable, the most commonly found situation in a kitchen in thecontext of assistive robotics, simple planning heuristic meth-ods based on assumption of a planar table can solve this lastissue [12]. These heuristic methods can be combined withpoint-to-point UVS for gross motion to the goal position andwith point-to-line or line-to-line control primitives for finealignment motions around the goal position. Of course ineach of these motions, the conditioning of the Jacobian J(t)will determine the precision of the visual servoing task.

The second ingredient for UVS seems to be availableand easy to implement (for example the Broyden updateapproach), thanks to extensive research and evaluation incontrol [10], [13], [14]. However the first one, which is thecrucial one, despite a lot of research [15], [16], [17], [18],is still hard to synthesize for reliable tracking of everydayobjects as shown by Fig. 1.

III. VISUAL TRACKING: KEY CHALLENGES ANDPOSSIBLE SOLUTIONS

Tracking is essential to visual servoing to know thelocation of the target object. Often this is done simply bytracking black and white markers on the robot and targetobject. However such a method will not work well in a realworld situation. We need to take into consideration otherapproaches to tracking. Familiar types of tracking algorithmsinclude segmentation based tracking, feature based trackingand registration based tracking. Any tracking algorithm used

Page 4: Towards Practical Visual Servoing in Roboticstatsambo/publications/TowardsPracticalVisualS… · Towards Practical Visual Servoing in Robotics Romeo Tatsambon Fomena Camilo Perez

for visual servoing needs to have good accuracy. In [19]Lieberknecht et al. compare the registration based efficientsecond-order minimization (E.S.M.) algorithm [17] to fea-ture based approaches, where E.S.M. generally has betterperformance. For normal textured scenes E.S.M. trackswithin two pixels accuracy. In this example registrationbased tracking does better than the feature based method.Below we will focus on registration based tracking.

In registration based tracking, also called sum of squaredintensity difference (SSD) tracking, we specify a smallimage template that surrounds the desired object in the firstimage frame. Then we want to be able to find the locationof this template in future frames. We warp incoming imageframes to align them with the original template using a warpfunction with a corresponding warp parameter. With eachnew image frame the goal is to update the warp parameter.The Lucas-Kanade algorithm [20] serves as a foundationfor many other registration based tracking algorithms. Byminimizing the SSD between the current image warped withthe latest warp parameter and the initial template, they areable to find the new update to the warp parameter. Thistechnique is fast and runs in real time.

Baker and Matthews [18] solve image registration basedtracking with their Inverse Compositional algorithm (I.C.)derived from the Lucas-Kanade algorithm. Benhimane andMalis E.S.M. algorithm [17] finds the same compositionalupdate by using an efficient second order minimization. Dicket al. [21] tries a different approach to registration basedtracking by introducing machine learning into their system.They use Nearest Neighbour Search with partially alignedtemplate images to find the corresponding warp parameterupdate. We have tried to replace the Nearest NeighbourSearch by a different machine learning approach to seeif the behaviour is similar. In this case we learned thefunction between the warp updates and partially alignedimage templates using Regularized Ridge Regression. Bothmachine learning approaches use a step of the InverseCompositional algorithm for fine alignments.

For registration based tracking to be useful to visual ser-voing, the algorithms need to be able to handle large imagemotion between subsequent frames. A common problemwith the Lucas-Kanade like approaches is that they onlyconverge for small image motion. Dick et al. inclusionof machine learning improved the convergence for largeimage motion. They implemented and tested the I.C., E.S.M.algorithms and their own Nearest Neighbour tracker (N.N.+ I.C.) on increasing static image motion. We repeat thesame experiment, but include the Ridge Regression (R.R. +I.C.) algorithm as well. Similarly to Dick et al. we used thefamous Lenna image for the experiment. Dick et al. [21]add random Gaussian noise with standard deviation tothe original object template image to create the updatedtemplate. The experiment is run for σ from 1 to 20. Eachsuch experiment is run 5000 times and records how often

the different trackers fail. The results are shown in Fig. 3.We can see that the Nearest Neighbour and Ridge Regressionalgorithms handle large image motion better than the InverseCompositional and EMS algorithms.

Figure 3. Tracker evaluation.

Tracking is just one component of an integrated visualservoing system which allows to select features for vi-sual servoing. Object recognition techniques can be usedto automously select features as in [3]. Different type offeatures can be found in the literature. For example thetrifocal tensor [22], the image and SIFT moments [23], [24],the cylindrical coordinates [25] and the spherical features[26], [27], [28]. Using the basic point and line features, thenext section presents the key challenges related to image-based task specification and a possible provable and intuitivespecification interface.

IV. VISUAL SPECIFICATION AND VERIFIABILITY: KEYCHALLENGES AND POSSIBLE PROVABLE AND INTUITIVE

SOLUTION

An example of a simple practical problem in visualservoing is the image task specification of the goal positionto reach. Most of the works carried out used the so-called”teach by showing” approach [4] to validate new controlconcepts. As shown in Fig. 4, this approach consists ofmoving the robot to the goal position and to record the goalvalue s∗ of the visual features before regulating the errore(t)= s(t)− s∗ to zero. The ”teach by showing” techniqueis a way to relate the image goal value s∗ to a consistentcamera pose oMc∗ ∈ SE(3) with respect to the objectframe. Although the teach by showing technique works wellfor repetitive tasks, it has slowed adoption of robotics incommon unstructured environments where the cameras arenot calibrated and where there is no full a priori knowledge

Page 5: Towards Practical Visual Servoing in Roboticstatsambo/publications/TowardsPracticalVisualS… · Towards Practical Visual Servoing in Robotics Romeo Tatsambon Fomena Camilo Perez

about the environment. This is simply due to the fact thatthe parameterization of SE(3) is not native to the visionsensor coordinate system. A path to solve this problemwas presented a long time ago in [29]. Here we proposethe use of an intuitive image-editor like solution for taskspecification. The user selects the type of alignment he/shewants to achieve between the point-to-point and point-to-line menus as shown in Fig. 5(b). Then he/she chooses thecorners of the target to track. Finally the control routineto zero the image errors, is launched. Fig 5(a) shows thedifferent steps of the control. Yet there is the challengingissue of mathematically specifying and verifying a SE(3)task into the image space.

Figure 4. Teach by showing.

Alignments and motions can be specified formally inSE(3) using 3D geometric constructions of 3D point fea-tures or 3D line features f , such as the 3D Cartesiancoordinates of a point in R3 or the Plucker coordinates ofa 3D line in the projective 3-space P3, on the object ofinterest and on the robot [30]. Briefly, a positioning goalcan be theoretically described by the constraint

T (f)= 0, (2)

where T : F → {0, 1} is a task function introducedin [31], with F the nF -dimensional manifold of all possibleconfigurations (nF ≤ 6). For example, to place a cup on atable, T would be a function which is zero when the pointson the bottom of the cup are in the plane of the table. Theactual coordinates of the points or lines f used to compute Tare unknown. Rather, the image projections of f are observedusing one or more cameras, yielding a vector of observationss ∈ S , with S the nS -dimensional manifold of all possibleconfigurations in the image space. In general, s is the onlydependable information available to perform an image-basedvisual servoing task. An image-based visual servoing task

(a)

(b)

Figure 5. Image-editor based task specification: (a) different steps of therealization of the task and (b) point-to-point and point-to-line alignmentinterface menus.

encoding can be represented by the constraint

E(s) = 0, (3)

where E : S → Rn is an image-based task functionas presented in [7].

A central issue is the decidability of a positioning taskfunction T on a given set C of camera models (formallydefined by the level of camera calibration). More precisely,the question is whether or not there exists an image-basedtask encoding function E for the positioning task functionT such that

∀C ∈ C, ∀f ∈ F , E(s) = 0 ⇐⇒ T (f) = 0, (4)

whereC : V → S

f 7→ s,

with V the visibility space such that V ⊆ P3(R), with P3(R)the real projective space of dimension 3. It has been shownthat, for a set Cproj of weakly calibrated stereo vision sys-tems, (4) holds when (2) is projectively invariant [30]2. For

2For example the midpoint task on a 1D line is undecidable, but thissame task is decidable on a 2D rectangle, i.e ”this cup is in the middle ofthe table” is decidable.

Page 6: Towards Practical Visual Servoing in Roboticstatsambo/publications/TowardsPracticalVisualS… · Towards Practical Visual Servoing in Robotics Romeo Tatsambon Fomena Camilo Perez

a set Cuncal of uncalibrated cameras, projective invarianceof the task T is a necessary but not sufficient condition foran image encoding E to exist.

The proofs cited above are non-constructive. However,there is strong evidence that on the set Cuncal, all decid-able tasks can be described as compositions of a smallset of basis tasks, where each ”task” defines geometricconstraints. Furthermore, it seems that there are other similartask classifications for different sets C of camera models(perfectly and weakly calibrated). We propose to researchan intuitive, yet complete set of basis tasks, the neededcomposition operators, and to implement them within avisual servoing framework. This is a completely new topicof research which needs to be addressed in order to meet thetask specification challenge in human-robot interaction in thepromising context of semi-autonomous control of robots.

Using the proposed approach, it is possible to improvethe intuitiveness of our human-robot interface by bringingit closer to the human-user. For instance by using a touch-screen or natural human gestures.

V. TOWARDS MORE INTUITIVE INTERFACES:TOUCSCREEN AND GESTURES

In addition to proving the realization of the task in theimage space, we also aim at reducing the cognitive load forthe human operator using the proposed interface. We do thisby using either a touchscreen or, simply by naturally pointingto a shared image of the object to grasp, in the case wherethe operator and the robot are located in different sites.After the object selection, visual information on the positionand orientation of the object are then used as feedback toautonomously move the robot close enough to the object tomanipulate it.

In the case where the human-operator and the robot arelocated in the same site (kitchen setting for example), thisgesture interface could be extended to pointing directly tothe object, in which case the human operator and the robotshare the same workspace [32].

A key issue in the design of interfaces is evaluation: whatshould be tested? How should this be tested? etc. Theseare some of the main questions which need to be answeredbefore proposing a visual servoing system to an end-user.Although our system has not been fully tested, we havevalidated our design in an application which consists ofsorting kitchen utensils from kid toys.

Using ROS framework, we have experimentally verifiedour solutions to these issues using our 7 DoFs Barrett WAMArm and a Kinect 3D sensor, a low cost vision system. Moreprecisely, we have designed and developed several intuitivevision-based control interfaces for a one-click-grasping ap-plication. Fig. 6 presents our touchscreen interface. Thisapplication can allow a handicapped or elderly person tograsp either a toy for his/her child or a kitchen utensil in

just one click. A video demonstrating our interface designis available at http://webdocs.cs.ualberta.ca/~vis/HRI/

Figure 6. Touchscreen task specification. Video demo is available athttp://webdocs.cs.ualberta.ca/~vis/HRI/

VI. CONCLUSION

In this paper we have proposed solutions towards practicalvisual servoing in robotics. The visual tracking problemwas revisited. We reviewed existing tracking methods andshowed some of their weakness in the context of visualservoing. The problem of task specification and verifiabilityhas been introduced as the main issue which restrict theapplication of this technique to solve real world problems.We have presented the challenges related to this issue andproposed a provable and intuitive image-editor interfacebased on composition of primitive tasks such as point-to-point and point-to-line alignment. We have recalled a simpleeffective solution to the path planning problem in the caseof a manipulation task on a table. Finally, we have shownpossible ways to improve the intuitiveness of our solutionusing either a touchscreen or simply pointing gestures on ashared image between the robot and the human operator.

Future works will focus on proving convergence of image-based composition of primitive tasks using our proposedimage-editor like interface. We will also focus on the soft-ware integration of the different components necessary toimplement a fully functional visual servoing system. Finally,as the main part of any systems development, we willextensively evaluate our system in real world conditions.

ACKNOWLEDGMENT

This work was supported by Natural Sciences and Engi-neering Research Council of Canada (NSERC) under theCollaborative Research and Development (CRD) Grantswith the Canadian Space Agency (CSA).

REFERENCES

[1] S. Edwards and C. Lewis, “Ros-industrial:applying the robotoperating system (ros) to industrial applications,” in IEEE Int.Conference on Robotics and Automation, ECHORD Work-shop, St Paul, Minnesota, May 2012.

Page 7: Towards Practical Visual Servoing in Roboticstatsambo/publications/TowardsPracticalVisualS… · Towards Practical Visual Servoing in Robotics Romeo Tatsambon Fomena Camilo Perez

[2] M. Quigley, B. Gerkey, K. Conley, J. Faust, T. Foote, J. Leibs,E. Berger, R. Wheeler, and A. Ng, “Ros: an open-sourcerobot operating system,” in IEEE Int. Conference on Roboticsand Automation, Workshop on Open Source Robotics, Kobe,Japan, May 2009.

[3] K. M. Tsui, D.-J. Kim, A. Behal, D. Kontak, and H. A. Yanco,“”i want that”: Human-in-the-loop control of a wheelchair-mounted robotic arm,” Journal of Applied Bionics and Biome-chanics, vol. 8, no. 1, pp. 127–147, Apr. 2011.

[4] S. Hutchinson, G. Hager, and P. Corke, “A tutorial on visualservo control,” IEEE Transactions on Robotics and Automa-tion, vol. 12, no. 3, pp. 651–670, Oct. 1996.

[5] F. Chaumette and S. Hutchinson, “Visual servo control, part i:Basic approaches,” IEEE Robotics and Automation Magazine,vol. 13, no. 4, pp. 82–90, December 2006.

[6] ——, “Visual servo control, part ii: Advanced approaches,”IEEE Robotics and Automation Magazine, vol. 14, no. 1, pp.109–118, March 2007.

[7] B. Espiau, F. Chaumette, and P. Rives, “A new approach tovisual servoing in robotics,” IEEE Transactions on Roboticsand Automation, vol. 8, no. 3, pp. 313–326, Jun. 1992.

[8] W. Wilson, C. Hulls, and G. Bell, “Relative end-effectorcontrol using cartesian position-based visual servoing,” IEEETransactions on Robotics and Automation, vol. 12, no. 5, pp.684–696, Oct. 1996.

[9] E. Malis and F. Chaumette, “Theoretical improvements in thestability analysis of a new class of model-free visual servoingmethods,” IEEE Transactions on Robotics and Automation,vol. 18, no. 2, pp. 176–186, Apr. 2002.

[10] M. Jagersand, O. Fuentes, and R. Nelson, “Experimental eval-uation of uncalibrated visual servoing for precision manipu-lation,” in IEEE Int. Conference on Robotics and Automation,1997.

[11] R. Tsai and R. Lenz, “A new technique for fully autonomousand efficient 3d robotics hand/eye calibration,” IEEE Trans-actions on Robotics and Automation, vol. 5, no. 3, pp. 345–358, June 1989.

[12] M. Jagersand, “On-line estimation of visual-motor modelsfor robot control and visual simulation,” Ph.D. dissertation,University of Rochester, 1997.

[13] A. Shademan, A.-M. Farahmand, and M. Jagersand, “Towardslearning robotic reaching and pointing: an uncalibrated visualservoing approach,” in Conference on Computer and RobotVision, 2009.

[14] A.-M. Farahmand, A. Shademan, M. Jagersand, and C. Szepe-vari, “Model-based and model-free reinforcement learning forvisual servoing,” in IEEE Int. Conference on Robotics andAutomation, 2009.

[15] R. Richa, R. Sznitman, R. Taylor, and G. Hager, “Visualtracking using the sum of conditional variance,” in IEEE/RSJInt. Conference on Intelligent Robots and Systems, sept. 2011,pp. 2953 –2958.

[16] G. D. Hager, M. Dewan, and C. V. Stewart, “Multiple kerneltracking with ssd,” in Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition, 2004.

[17] S. Benhimane and E. Malis, “Real-time image-based track-ing of planes using efficient second-order minimization,” inIEEE/RSJ Int. Conference on Intelligent Robots and Systems,vol. 1, sept.-2 oct. 2004, pp. 943 – 948.

[18] S. Baker and E. Matthews, “Equivalence and efficiency of im-age alignment algorithms,” in IEEE Conference on ComputerVision and Pattern Recognition, 2001.

[19] S. Lieberknecht, S. Benhimane, P. Meier, and N. Navab,“A dataset and evaluation methodology for template-basedtracking algorithms,” in 8th IEEE International Symposiumon Mixed and Augmented Reality, Oct. 2009.

[20] S. Baker and I. Matthews, “Lucas-kanade 20 years on: Aunifying framework: Part 1.” CMU-RI-TR-02-16, RoboticsInstitute, Carnegie Mellon University, Tech. Rep., 2002.

[21] T. Dick, C. Perez, A. Shademan, and M. Jagersang, “A super-vized learning approach to registration-based visual tracking,”in Submitted to Robotics: Science and Systems, 2013.

[22] A. Shademan and M. Jagersand, “Three-view uncalibratedvisual servoing,” in IEEE/RSJ Int. Conference on IntelligentRobots and Systems 2010, oct. 2010, pp. 6234 –6239.

[23] O. Tahri and F. Chaumette, “Point-based and region-basedimage moments for visual servoing of planar objects,” IEEETransactions on Robotics, vol. 21, no. 6, pp. 1116–1127,December 2005.

[24] F. Hoffmann, T. Nierobisch, T. Seyffarth, and G. Rudolph,“Visual servoing with moments of sift features,” in IEEEInternational Conference on Systems, Man and Cybernetics,vol. 5, Oct. 2006, pp. 4262 –4267.

[25] M. Iwatsuki and N. Okiyama, “A new formulation of vi-sual servoing based on cylindrical coordinate system,” IEEETransactions on Robotics, vol. 21, no. 2, pp. 266 – 273, April2005.

[26] R. Tatsambon Fomena, O. Tahri, and F. Chaumette,“Distance-based and orientation-based visual servoing fromthree points,” IEEE Transactions on Robotics, vol. 27, no. 2,pp. 256–267, April 2011.

[27] P. Corke, “Spherical image-based visual servo and structureestimation,” in IEEE Int. Conference on Robotics and Au-tomation, May 2010, pp. 5550 –5555.

[28] R. Tatsambon Fomena and F. Chaumette, “Improvementson visual servoing from spherical targets using a sphericalprojection model,” IEEE Transactions on Robotics, vol. 25,no. 4, pp. 874–886, August 2009.

[29] G. Hager, “A modular system for robust positioning usingfeedback from stereo vision,” IEEE Transactions on Roboticsand Automation, vol. 13, no. 4, pp. 582–595, Aug.

[30] Z. B. Dodds, “Task specification languages for uncalibratedvisual servoing,” Ph.D. dissertation, Yale University, 2000.

Page 8: Towards Practical Visual Servoing in Roboticstatsambo/publications/TowardsPracticalVisualS… · Towards Practical Visual Servoing in Robotics Romeo Tatsambon Fomena Camilo Perez

[31] C. Samson, M. Le Borgne, and B. Espiau, Robot control: theTask Function approach. Oxford University Press, 1991.

[32] C. P. Quintero, R. T. Fomena, A. Shademan, N. Wolleb,T. Dick, and M. Jagersand, “Sepo: Selecting by pointing asan intuitive human-robot command interface,” in IEEE Int.Conference of Robotics and Automation, Karslruhe, Germany,May 2013.