Pretouch Sensing for ManipulationPretouch Sensing for Manipulation Liang-Ting Jiang Department of Mechanical Engineering University of Washington Seattle, WA 98105 Email: [email protected]

Pretouch Sensing for ManipulationLiang-Ting Jiang

Department of Mechanical EngineeringUniversity of Washington

Seattle, WA 98105Email: [email protected]

Joshua R. SmithDepartments of Computer Science and Engineering

Department of Electrical EngineeringUniversity of Washington

Seattle, WA 98105Email: [email protected]

Abstract—“Pretouch” refers to sensing modalities that areintermediate in range between touch and vision. This paperdescribes two pretouch senses we have created (electric field andseashell effect), and discusses the use of pretouch in manipulation.

I. INTRODUCTION

Our work on “pretouch” sensing is exploring the hypothesisthat short range, non-contact sensors mounted in robot handscan benefit manipulation. In this paper we review two pretouchsensing mechanisms we have created, discuss their applicationin manipulation tasks, and outline future work using pretouchfor manipulation. 1

A. Why Pretouch for Manipulation?

The range at which vision operates is essentially unbounded.Thus vision is a natural sense to rely on for detecting andlocalizing objects of interest at long range. At short range,however, when it comes time to execute a grasp, vision hasshortcomings. The robot’s hand and arm often occudes a head-mounted camera’s view of the object to be manipulated. Ahead-mounted camera is not in the same coordinate frame asthe hand, and thus even if the camera has an unrestricted viewof the object, uncertainties associated with actuation can leadto manipulation errors.

Touch sensing occurs in the frame of the hand, and is notsubject to occlusion, but because it relies on contact betweenthe manipulator and the object, touch sensing tends to displaceobjects whose positions are not precisely known in the hand’scoordinate frame.

The aim of pretouch sensing is to combine the desirablefeatures of these two sensing modalities: like touch, it providesinformation about the relative geometry of the hand and theobject, in the hand’s coordinate frame; like vision, it is non-contact and does not disturb the object’s position.

II. ELECTRIC FIELD PRETOUCH

1) Physics of Electric Field Sensing: In Electric FieldSensing, an AC signal is applied to a transmit electrode.This induces an AC current in the receive electrode, whichis amplified and processed by the analog front end (a currentamplifier, which measures current induced at the receiver) andsubsequent signal processing (in our case, an analog to digital

1This paper is an overview extracted from our previous papers [1][2].

Mid-Range Long-Range

Fig. 1. Iso-signal surfaces illustrating response of the electric field pretouchsensor.

converter and signal processing software in a microcontroller).The sensed object modifies the current induced in the readerby interacting with the transmit and receive antennas.

In shunt mode (the most commonly used mode), the objectis well-grounded. Bringing a sensed object closer to transmit-receive pair shunts displacement current that would haveotherwise reached the receiver, decreasing the measured sensorvalue as the object gets closer to the electrodes. Figure 1 showsthe response of the sensors to a small grounded object; theobject placed anywhere on the pictured iso-signal surfaceswill yield the same sensor reading. The sensor range isdetermined by the transmit-receive electrode spacing. Largerspacing results in a sensor that operates on a longer lengthscale. Figure 1 compares mid-range and long-range sensors.

Note that at the frequencies at which we are operating, thehuman is typically well-coupled to ground, often through theshoes. (We are in the regime of AC coupling, so although thereis typically no DC electrical path from your body to ground,there is usually a relatively good high-frequency AC path toground.) This in turn means that conductive objects that aperson holds or touches are also relatively well-grounded.

The sensors detect both conductive objects, and non-conductive objects whose dielectric constants differ from thatof the air. Only the surface of a conductive object affects thesensors. For dielectrics, however, the entire bulk of the materialaffects the sensors. For this reason, the net dielectric constantof an object is proportional to density. Fig. 3 compares theresponse of various dielectric objects to a conductor. Some ofthe dielectric objects work quite well. The ones that do notare very low density.

Left Receive

Short RangeTransmit

Mid-RangeTransmitRight Receive

Fig. 2. Electric field pretouch sensor hardware, designed for mounting inthe fingers of the Barrett Hand.

1 2 3 4 5 6−0.5

0

0.5

1

1.5

2

Nor

mal

ized

Sen

sor R

espo

nse

E−field Sensor Response to Non−conductive Objects

Long Range SensorMid Range Sensor

CopperSphere

Ceramic Mug

ShotGlass

AcrylicBlock

WoodBlock

EPSFoamBlock

Fig. 3. Response of electric field sensors to non-conductive materials(dielectrics). For dielectric objects, sensor response is typically proportionalto object density, which could be another useful pre-touch cue.

III. SEASHELL EFFECT PRETOUCH SENSOR

Optical and electric field sensors both rely on electro-magnetics (though at greatly different wavelengths). It wouldbe desirable to have a pretouch modality that worked usingcompletely different physical mechanisms, to increase thechances that a second technique would detect the material ifthe first failed.

Acoustic impedance is a convenient measurement of howmuch the particles move given the sound pressure at particularfrequencies. At the open end of a pipe, the impedance issupposed to be zero (infinite velocity), but the sound fieldaround the opening results in additional radiation impedance.The imaginary part of the radiation impedance introducesan end correction to the original pipe length, therefore theresonance frequency of the pipe is changed. When an objectapproaches to the pipe opening, the sound field betweenthe surface of the object and the pipe opening will furtherintroduce more length correction to the pipe. Dalmont [3]found an empircal formula for the end correction in this kindof pipe-surface configuration by fitting a function to valuesproduced by a finite element model.

A. Sensor DesignIn order to detect changes in pipe resonance, a microphone

collects the sound filtered by the acoustic cavity (i.e. thepipe). The pipe’s resonant frequency is found by lookingfor a peak in the spectrum. To avoid being confounded byfeatures in the raw (unfiltered) ambient audio itself (includingloud ambient sounds), a reference microphone is used tocollect environmental sounds not filtered by the pipe; thisbackground spectrum is subtracted from the actual pretouchsensor channel. This noise cancellation approach substantiallyimproves sensing accuracy.

Unlike the previous sensor [1], the new sensor described inthis paper is completely integrated into the Willow GaragePR2 robot; all external cables and electronics have beeneliminated. A customized printed circuit board (PCB) andfingertip structure was designed to hold all the electronic andmechanical components, including the microphone and thepipe. Figure 2(b) shows the PCB and fingertip fixture with allthe components attached, and Figure 2(a) shows the completedpretouch sensing fingertip installed on the Willow Garage PR2gripper. In the current implementation, we have two fingerson one of the grippers as the actual sensing channels, and areference channel on another gripper for spectrum subtraction.The only difference between the design of the actual sensingchannels and the reference channel is that the microphoneon the reference channel is not attched to the pipe (acousticcavity), so it simply collects the ambient (unfiltered) sound.

The sound signal path implementation is decribed brieflyhere. The pipe cavity filters the ambient noise, and the soundsignal collected by an electret microphone and amplified by a40dB-Gain low-noise amplifier.

The power spectral density of the sound signal from bothchannels are estimated using Welch spectrum estimation (Ns =1024; overlap ratio = 70%; Hanning data taper). The spectrumof the reference channel is subtracted from the spectrumof the sensor signal before peak finding, which avoids theeffect of loud sounds, outside of the sensor’s frequency range,misleading the peak tracking. The peak finding and estimationalgorithm we used here is the same as described in [1], whichis adapted from [4].

The new embedded sensor design eliminates the constraintson robot arm motion caused by the external wires and elec-tronics, and thus broadens the applicability of the sensors. Anobject contour tracking application enabled by the new sensordesign will be described later. The design presented here couldbe adpated to integrate the sensor into other robotic platforms.

B. EvaluationThe performance of the sensor on the fingertip is evaluated

by collecting 1000 sensor readings (filtered spectral peakfrequency) at various distance from 1mm to 10mm. A box-and-whisker plot is plotted to present the performance ofthe sensor (Figure 5). A contrast to noise ratio (CNR) forevaluating the sensor performance is defined as

CNR =mean(f10)−mean(f1)

mean(std(fi)), i = 1..10

(a)

(b)

MicrocontrollerMicrophone & Sensor Cavity

Microphone Amplifier

SPI / Programmer

Fig. 4. The seashell effect pretouch sensor (a) mounted in the PR2 robot;(b) sensor hardware

where i is the distance in millimeter. The box plot showsthe filtered resonance frequency starts to decrease from 6mm. Based on that, we select the threshold to 9500Hz (thelower quartile at 3 mm) as the first distance being able tobe measured with confidence. In this way, the upper quartileat 3 mm is smaller than the lower quartile at 6 mm, so thesensor is not getting confused. All the software in this workis implmented in ROS (Robot Operation System). The signalprocessing and frequency estimation is implemented in Pythonas a ROS node, which continuously publishes the estimatedresonance frequency, total signal energy, and distance in a ROSmessage at the rate of 20 Hz.

C. Material sensitivity

The seashell effect pretouch sensor does not depend onoptical or electrical material properties. Instead, it depends onmechanical / acoustic properties. This characteristic makes ita good complement to long range optical depth sensors. Forexample, seashell effect pretouch can sense highly transparent,reflective, or light-absorbing materials, which are difficult foroptical sensors. In our previous work, we reported that theonly materials we have found on which the sensor fails areopen foams and certain rough fabrics. In this section, wecompare the sensor output for several different materials at thesame distance. 1000 readings are measured for each object at

1 2 3 4 5 6 7 8 9 10Distance (mm)

7500

8000

8500

9000

9500

10000

10500

11000

Res

onan

ceFr

eque

ncy

(Hz)

Contrast to Noise Ratio (CNR) = 22.04

Pretouch Thresold

Fig. 5. Performance of the seashell effect sensor

Cloth Foam Glass Cup

IronPlate

Paper Box

PVC Plastic

TissuePaper

Fig. 6. The material sensitivity of the seashell effect pretouch sensor.

a distance of 2mm from the object’s surface. The collecteddata is plotted in 6. The results show that the readings frommaterials with more porosity, such as cloth and foam, aremore noisy than others. We hypothesize that these materialsare the acoustic analog of optically transparent and absorbtivematerials, respectively.

IV. APPLICATIONS OF PRETOUCH

A. Finger alignment and grasp pre-shaping

The Barrett Hand is a multi-fingered gripper in which eachfinger can be actuated independently. We used electric fieldpretouch to independently servo each finger to be close to, butnot in contact with, the object. Then by decreasing the servoingsetpoint, the 3 fingers approach the object in a coordinatedfashion. The fingers make contact with the object at nearlythe same time, which typically allows all 3 fingers to makecontact without displacing the object.

Fig. 7. (a) PR2 robot probing the unseen area of the bottle and addingpretouch pointcloud before grasp planning. (b) The 3-D visualization in rviz.The red points are generated from the narrow stereo camera , and the yellowpoints are generated by the pretouch sensor during the pretouch motion. (c)The view from the left-narrow stereo camera overlayed with the visualizationmarkers.

B. Arm alignment and servoing

In another application of electric field pretouch, we trans-mitted from the palm of the Barrett hand to receivers in eachfinger tip. By moving the endpoint of the arm within a pre-selectred plane until the 3 sensor values balanced, we wereable to servo the arm to align with an object presented by ahuman.

C. Reactive grasping of compliant objects

In one of our first demonstrations of seashell effect pretouch,we showed that the seashell effect sensor can detect highlycompliant and insubtantial objects, such as thin paper boxes.This object cannot be reliably detected by e-field sensors,and is too insubstantial for the PR2’s grip-mounted pressuresensors to detect, leading to crushing errors.

D. Pretouch assisted grasp planning

In this application, the pretouch sensor is combined withinformation from arm kinematics to augment a partial point-cloud created by a depth sensor.

V. RELATED WORK

Hsiao et al. [5] described an optical pretouch system, withoptical emitters and detectors built into the fingers of a BarrettHand.

Capacitive sensing has been explored in robotics in variouscontexts. In addition to our prior work, recently Solberg,Lynch, and MacIver presented fish-inspired underwater robotcapable of localizing object using electric field sensing. [6]For several earlier instances of above-water capacitive sensingfor robotics, please see [7], [8], [9] and [10]. None of theseschemes were targeted specifically at sensing for manipulation.

We are not aware of any prior work on seashell effectsensing.

VI. FUTURE WORK

One promising future direction is to combine multipledistinct pretouch modalities.

We plan to integrate pretouch sensing more deeply withgrasping. In [1], we demonstrated that the pretouch sensorcombined with information from the arm encoders can beused to generate additional 3D points, augmenting a parialpointcloud collected by a Kinect operating on a partiallytransparent object. One next step is to choose pretouch probepoints in a sophisticated fashion, perhaps by probing regions ofmaximum uncertainty. Beyond that, we aim to truly integrategrasping and sensing. A grasp plan would be generated usingpartial information from a depth sensor. As the grasp plan isexecuted, additional information collected by the pretouch sen-sors would be used to refine and replan the grasp. This effortwill require breaking abstraction barriers between normallyseparate processes of sensing, perception, and manipulation.

REFERENCES

[1] L. Jiang and J. Smmith, “Seashell effect pretouch sensing for roboticgrasping,” in IEEE International Conference on Robotics and Automa-tion (ICRA ’12), 2012.

[2] B. Mayton, L. LeGrand, and J. Smith, “An electric field pretouch systemfor grasping and co-manipulation,” in IEEE International Conference onRobotics and Automation (ICRA ’10), 2010.

[3] J. Dalmont, C. Nederveen, and N. Joly, “Radiation impedance oftubes with different flanges: numerical and experimental investigations,”Journal of Sound and Vibration, vol. 244(3), pp. 505–534, 2001.

[4] E. Jacobsen and P. Kootsookos, “Fast, accurate frequency estimators,”IEEE Signal Processing Magazine, vol. 24(3), pp. 123–125, 2007.

[5] K. Hsiao, P. Nangeroni, M. Huber, A. Saxena, and A. Y. Ng, “Reactivegrasping using optical proximity sensors,” in To appear in InternationalConference on Robotics and Automation (ICRA), Kobe, Japan, 2009.

[6] J. R. Solberg, K. M. Lynch, and M. A. MacIver, “Robotic electrolo-cation: Active underwater object localization with electric fields,” inIEEE International Conference on Robotics and Automation, Rome,Italy, 2007.

[7] G. Mauer, “An end-effecter based imaging proximity sensor,” Journalof Robotic Systems, pp. 301–316, 1989.

[8] J. Novak and I. Feddema, “A capacitance-based proximity sensor forwhole arm obstacle avoidance,” in Proceedings of the 1992 IEEEInternational Conference on Robotics and Automation, Nice, France,1992, pp. 1307–1314.

[9] N. Karlsson, “Theory and application of a capacitive sensor for safe-guarding in industry,” in Proceedings of IEEE Instr. and Msmt. Tech-nology Conf.–IMTC 94, Hammamatsu, Japan, 1994.

[10] D. Schmitt, J. Novak, G. Starr, and J. Maslakowski, “Real-time seamtracking for rocket thrust chamber manufacturing,” in IEEE Roboticsand Automation Proceedings, 1994.