The “Silk Cursor”: investigating transparency for 3D target acquisition

Boston,MassachusettsUSA* April24-28,1994%?

HumanFactorsinComputigSystems

The “Silk Cursor”:Investigating Transparency for 3D Target Acquisition

Shumin Zhail William Buxton2 Paul kiilgram]”

lDepartment of Industrial EngineeringUniversity of Toronto

Toronto, ontario, M5S 1A4, CanadaTel: 1-416-978-3776

E-mail: [email protected]. edumilgram@ ie.utoronto. ca

ABSTRACT

This study investigates dynamic 3D target acquisition. Thefocus is on the relative effect of specific perceptual cues. Anovel technique is introduced and we report on anexperiment that evaluates its effectiveness.

There are two aspects to the new technique. First, incontrast to normal practice, the tracking symbol is avolume rather than a point. Second, the surface of thisvolume is semi-transparent, thereby affording occlusioncues during target acquisition.

The experiment shows that the volumehcclusion cues wereeffective in both monocular and stereoscopic conditions.For some tasks where stereoscopic presentation isunavailable or infeasible, the new techniaue offers aneffective alternative.

KEYWORDS: 3D interface, interaction technique,acquisition, virtual reality, Fitts’ law, input,perception.

INTRODUCTION

targetdepth

With the advent of modem workstations and increasingdemands for computer-based applications, 3D techniques aremoving from the restricted domain of graphics tomainstream applications (e.g. [2]). However, as we move to3D, we see a breakdown in many of the interactiontechniques that have traditionally been used in 2D directmanipulation systems. Tasks such as inking, targetacquisition, pursuit tracking, sweeping out regions,orientation, navigation and docking present new challengesto the interaction designer.

Largely as an outgrowth of computer graphics, a body of

*Cumentty on leave at ATR Communication SystemResearchLaboratories,Kyoto, Japan

Permission to copy without fee all or part of this material is

granted provided that the copies are not made or distributed fordirect commer “al advantage, the ACM copyright notice and tha

Jtitle of the p Iication and its date appear, and notice is given

that copying is by permission of the Association for Computing

Machinery. To copy otherwise, or to republish, requires a fee

and/or specific permission.

CH194-4/94 Boston, Massachusetts USA

e 1994 ACM 0-89791 -650-6 /94/0459 . ..$3.50

2University of Toronto & Xerox PARCc/o CSRI

University of TorontoToronto, ontario, M5S 1A4, Canada

Tel: 1-416-978-1961E-mail: buxton @dgp.toronto. edu

research is developing which is beginning to address someof the interaction issues confronting the designer.Representative examples are found in [4, 8, 11, 131.However, there remain large gaps both in the literature andin practice, and of the techniques described, there has beenlittle in the way of experimental evaluation. Chen,Mountford & Sellen [3] is one of a few notable exceptions.

In the study reported below, our intent is to contribute tothis body of research. We introduce a new technique fordynamic 3D target acquisition. After describing thetechnique, we report on an experiment that evaluates itseffectiveness under various circumstances.

The technique described is novel in two respects. First, thetracking symbol used is a volume rather than a point, as isthe case in conventional systems. Second, the surface ofthe tracking volume is semi-transparent, thereby providingadditional depth cues beyond what is achievable withconventional techniques, primarily due to partial occlusionof the tracking volume by the object being tracked.

After a brief discussion of these two novel aspects of ourdesign, we proceed to describe the experiment in which thetechnique which uses them was tested.

The “Prince” Technique: the Cursor as Region orVolume

One of the most studied aspects of HCI is target acquisitionusing Fitts’ Law [5]. According to this law, the movementtime between two targets of width “W,” separated byamplitude “A”, can be modelled as follows [7]:

MT= a + b log2(A /W+ 1)

This is illustrated in Fig. l(a).

In this paper, we introduce a variation on the conditions towhich this model pertains. As is illustrated in Fig. l(b),we have reversed the situation such that the objects beingselected are points (separated by distance “A”), and thecursor is a region of width” W“.

459

m HumanFactorsinComputimgSystems (III ’94e “CelebratingInterdependence”L.LL L’

Our assumption is that Fitts’ Law holds under this newcondition. We have dubbed this the “Prince” technique,after the first company to make over-size tennis rackets(which epitomize the underlying principle).

a

b

El + n-w- --4~1-

u

I 4 vvt- 1Figure 1. Two representations of Fitts’ Law. The tophalf (a) shows the traditional representation. Targetsof width “W” are selected by the cursor (the pointdefined by the “+”), across amplitude “A. In the lowerhalf (b), two points (represented by the two “+”symbols), separated by amplitude “A are selected by acursor of width “W”.

We believe the idea of using a region for the cursor hasvalue in cases where one is selecting individual points, orsmall objects, or collections of points. We also believe thatthe concept will extend to 3D by having the cursor be avolume.

The “Silk Stocking” Effect: Using Occlusion forEnhanced Depth Cues

The second novel aspect of the technique which weintroduce is our proposed means for obtaining depth cuesthrough the use of occlusion. The simplest way to describeour technique is to imagine that the surface of the cursor’svolume is covered by a nearly transparent material, like asilk stocking. This is illustrated in Fig. 2.

Volume Cursor

Figure 2: Using a “silk” covering over a rectangularvolume cursor in order to obtain occlusion-based depthcues. An object at point A is seen through two layersof “silk”, and so is perceived to be behind the volumecursor. An object at point B is seen through one layer,and so is perceived as inside the cursor’s volume. Anobject at point C is not occluded by the silk at all, andso is seen to be in front of the volume.

Using this technique, which has recently been enabled bythe power of modem graphics workstations, one can easily

tell if an object is behind, inside or in front of the cursor.(seeColour plates).

EXPERIMENT

Experimental Hypothesis

The primary goal of our experiment was to evaluate theeffectiveness of the silk surface on a volume cursor in a 3Ddynamic target acquisition task. We tested the volumecursor both with the silk surface and without it (i.e., in anoutline “wire frame” version). Since stereoscopic projectionis widely recognised as one of the most effective andcommon 3D interface techniques[12, 14, 15], we tested eachin both mono and stereo display conditions. Thus, theexperiment had four conditions: stereo display with silkcursor (SS), stereo display with wire frame cursor (SW),mono display with silk cursor (MS) and mono display withwire frame cursor (MW).

Our hypothesis was that the silk-like surface and thestereoscopic display would each significantly improve 3Dtarget acquisition, and that the two factors together wouldenhance each other. What was of particular interest to uswas whether or not the silk surface effect alone (i.e. the MScondition) would generate superior, or in any casecomparable, performance to the SW condition, which wouldconfirm to us the @tential advantages of the silk cursor onits own as a 3D target acquisition technique.

Experimental Task

A 3D dynamic target acquisition task, “virtual fishing”, wasdesigned for the experiment. In each trial of the experiment,an “angel fish” with random size and color appearsswimming around randomly within a 3D virtualenvironment, as shown in Fig. 3.

Figure 3: The “fishing” task

The subjects were asked to move a 3D volume cursor toenvelop the fish and “grasp it” when the fish was perceivedto be completely inside of the cursor. Subjects wore aspecial glove, and “grasping” was done by closing the hand.

460

Boston,MassachusetkUSA~ April24-28,1994 HumanFactorsinComputiigSystems!%?

If the fish was entirely inside of the cursor volume, the trialwas successful and the fish stayed “caught” inside of thecursor. The time score of the trials was displayed to thesubject, along with a short beep. If the fish was notcompletely inside of the cursor when grasped, the fishdisappeared. In this case, which was considered a “miss”, along beep was sounded and error magnitude in each x, y,and z dimension was displayed. Subjects pressed thespacebar on the workstation to activate each new trial.

The origin of the {x y z} coordinate system was located atthe center of the computer screen surface, with positive xaxis pointing to right, y pointing up and z pointing to theuser. All objects were drawn using polar projection andwere modelled in units of centimeters, where 1 cm in thevirtual fish tank corresponded to 1 cm in the real world forany line segment appeming within the same plane as thesurface of the screen. The x (from lips to tail end), y(vertical) and z (from left fin tip to right fin tip) dimensionsof the largest “adult” fish were 10 cm, 15 cm and 1.3 cmrespectively. The smallest “baby” fish was 30 percent of thesize of the largest “adult” fish. The cursor had a constantsize of 11.3 cm, 16.3 cm and 2.6 cm in x, y and zdimensions.

The fish movement was driven by independent forcingfunctions in the x , y and z dimension. Since a suitablecombination of sine functions generates smoothsubjectively unpredictable motion, it has beenconventionally used in manual tracking research [9]. in thisexperiment, forcing functions applied to the fish motionwere

x(t) = ~ Ap-i sin(2zfopit + $x (i))i=i)

y(t) = ~Ap-i sin(2zfopit + @y(i))i=o

z(t) = –7.8+ ~Ap-i sin(2n#opit+ @z(i))i=l)

where t is time. A = 4.55 cm, p = 2, f. = 0.02 Hz; @x(i)

@Y(i) and @z(i) are pseudo-random numbers, ranging

uniformly between O and 27c. (For a more detailedexplanation of these forcing functions, see [17]).

In silk cursor mode, the semi-transparent surface intensity Iwas rendered by interpolating the cursor colour intensity(source) Is with the destination colour intensity Id [6]

according toI=(x~+(l - rx)lij

The u coefficient was set at 0.38 for all surfaces of thecursor, except the back surface for which u was set at 0.6.

Experiment Platform

The experiment was conducted using the MITS(Manipulation in Three Space) system [16, 17] developed

by the authors. MITS is a non-immersive stereoscopicvirtual environment, based on a SGI IRIS 4DCrimson/VGX graphics workstation equipped withCrystalEyesTM stereoscopic glasses. In this experimen~ thecursor was driven by a self designed glove based on an

Ascension Technology BirdTM. Only translations wereinvolved in the fishing task. The graphics update rate wascontrolled at 15 Hz.

Experimental Design and Procedure

Eleven male and one female paid volunteers served assubjects in this experiment. The subjects were screenedthrough the Bausch & Lomb Orthorator visual acuity andstereopsis tests. Subjects ages ranged from 18 to 36, withthe majority in their early and mid-20’s. One of the 12subjects was left handed and the rest were right handed, asdetermined by the Edinburgh inventory. Subjects were askedto wear the input glove on their dominant hand.

A balanced within subjects design was used. The 12subjects were randomly assigned to a unique order of thefour conditions (SS, SW, MS, MW) by a hyper-Graeco-Latin square pattern, which resulted in every conditionbeing presented an equal number of times as fiist, second,third and finat condition.

Following a 2 minute demonstration of all experimentalconditions, the experiments with each subject were dividedinto four sessions, with one experimental condition in eachsession. There was a 1 minute rest between every twosessions. Each session comprised 5 tests. Test 1 startedwhen the subject had no experience with the particulwexperimental condition. Test 2, 3, 4, and 5 started after thesubjects had 3, 6, 9 and 12 minutes experience respective y.Practice trials occurred between the tests. Each test had 15trials of fish catching. At the end of each test, the numberfish both caught and missed (as both an absolute numberand a relative percentage) and mean trial time were displayedto the subject.

At the end of the experiment, a short questionnaire wasconducted to colleci users’ subjective preferences for allexperimental conditions.

Performance Measures

Task performance was measured by trial completion time,error rate (capture and miss percentage) and error magnitude.Trial completion time was defined as the time duration fromthe beginning of the trial to the moment when the subjectgrasped. Error rate was defined as the percentage of fishmissed in a test. Whenever a fish was missed, the errormagnitude was defined as the Euclidean summation of x, y,z errors:

Error Magnitude = t~

Experimental Results

In total our experiment, which comprised 3600 experi-mental trials (i.e. 2 (cursor types) x 2 (display modes) x 12

461

52!!HumanFactorsinCompufingSystems CHI’940 “Celebratinglnteu!ependence”

(subjects) x 5 (tests) x 15 (trials per test)), with 3performance measures per trial (i.e. trial completion time,error rate, and error magnitude), yielded 10,800 data points.Linear variance analysis was used to evaluate the statisticalsignificance of the independent variables and their potentialinteractions for each of the three performance measures.Logarithmic non-linear transformations were applied tocompletion time and error magnitude scores in analyzingstatistical significance, since residual analysis showed thatthese two measures were skewed towards the short fittedvalues. This section presents the primary results of thestatistical analysis.

Tria/ Completion Time. The variance analysis indicated thatcursor type (F(l, 3567) = 1148.5, p<.0001), display mode(F(l, 3567) = 630.3, p<.0001), cursor type and displaymode interaction (F(l, 3567) = 253.5, p<.0001), subjects(F(l 1, 3567) = 96.1, p<.0001), learning phase (F(4, 3567)= 70.1, p<.0001) and triat number (different fish size) allvery significantly affected trial completion time. Fig. 4illustrates the effect of cursor type (silk vs. wire frame) anddisplay mode (stereo vs. mono) to completion time.

4.5

: /“

H Mono

3.5 0 Stereo

3

‘; /

Sw

MSs

Silk Wireframe

Figure 4 Trial completion time performance in relationto cursor type and display mode

Ranking these results in the order from best to worst, themean completion time for each of the four interfaces wereas follows. SS: 2.09 sec.; MS: 2.38 sec.; SW: 2.90 sec.;MW: 4.61 sec. Post hoc analysis shows that the differencesbetween every pair of interfaces were significan~ all at the p<.0001 level.

Error Rate. The statistically significant factors affectingerror rate were: cursor type (F(l, 221) = 122.1, p<.0001),display mode (F(l, 221) = 67.9, p<.0001), cursor type anddisplay mode interaction (F(l, 221) = 33.0, p<.0001),subjects (F(l 1, 221) = 5.75, p<.0001), and learaing phaseF(4, 221) = 3.69, p = 0.0062). Fig. 5 illustrates the effectof cursor type and display mode on error rate.

The performance order of the four interfaces measured interms of error rate was exactly the same as that measured bytrial completion time. The mean error rate for eachcombination of the four interfaces were SS: 12.7%, MS:16.1%, SW: 20%, MW: 39.3%. The results of the Post-hoc pairwise comparison on error rate were as follows. SSVS. MS: p = .0795; SS’ VS. SW: p = .0002; SS VS. MW: p< .()()o~; MS VS. SW: p = .0479; MS VS. MW: p < .0001;

SW vs. MW: p <.0001. The statistical significance of thedifference between SS and MS was rather weak; however,

pairwise differences were significant to greater

40

35

30

; /

o25

‘: ::>10

Silk Wireframe

Mvv

Mono

Stereo

Sw

Figure 5: Error rate in relation to cursor type anddisplay mode

Error Magnitude. The effect of cursor type and display mode

on error magnitude are shown in Fig. 6. Error magnitudewas significantly affected by cursor type (Ill, 761) = 19.9,p<.0001), display mode (F(l, 761) = 39.2, p<.0001),subjects (F(l 1, 761) = 3.60, p<.0001), and experimentalphase (F(4, 761) = 3.88, p = 0.004). No significance forcursor type and display mode interaction (F(l, 761) = .009,p = .92) was foundjho”wever.

1.0 Stereo

Silk Wi;eframe

Figure 6: Error magnitude in relation to cursor type anddisplay mode

The means of error magnitude were 0.197, 0,337, 0.577,and 0.68 for SS, SW, MS, and MW interfaces respectively.Note that, in contrast to the other two error measures, SWproduced smaller errors than MS; however, the pairwisedifference between SW and MS was not statisticallysignificant (p = 0:16). All other pairwise differences weresignificant (from p = .003 to p < .0001).

Learn/ng Effects. Fig. 7 demonstrates subjects’completion time performances in relation to the learningphase. It shows that the relative scores between theinterfaces were consistent over the experimental tests.Subjects improved their time score in SS, MS and SWmode as they gained more experience, and presumably more

462

Boston,MassachusettsUSAo April24-28,1994 HumanFactorsinComputingSystemsR.——.

confidence. Little improvement in completion time wasfound with the MW condition.

testl test2 test3 test4 test5

Figure 7: Time performance with four interfaces at eachlearning phase

Fig. 8 gives error rate in relation to learning phase. Againthe relative rank of each mode was consistent across alt fivephases of the experiment. Interestingly, error rate for theMW condition showed the most obvious improvement overthe experiment. A small amount of improvement wasfound in the MS condition, and essentially none in the SSand SW modes.

Figure 8: Error rate with four interfaces at each learningphase

Comparing Fig. 7 with Fig. 8 reveals importantinformation about speed accuracy tradeoff patterns withrespect to learning. For the MW mode, subjects had more

than a 35% error rate, which apparently caused them tofocus on improving the accuracy aspect of the task at theexpense of time performance. In the other three cases (SS,MS, and SW), subjects already had less than a 25% errorrate and it appeafs that they were more satisfied with thislevel of accuracy, and thus were devoting more effort toshorting their trial completion times.

Subjective Preference. Subjective evaluations (Table 1) areconsistent with the other performance measures. SS was themost preferred and MW was the least liked. Of special

interest is the fact that MS was ranked higher than SW.

Summary of Resu/ts. The experiment largely confirmed ourinitial hypothesis. In general, the “silk surface” was the

most effective factor for successful acquisition. While stereopresentation in combination with the silk surface improvedperformance significantly, performance with the silk surfacein mono display mode was in fact better than the stereowire frame case for all measures except error magnitude, forwhich no significant difference was found.

SubjectivePreferences

Very Low OK High VeryLow High

MW 8 3 1

fvts 4 4 4

Sw 2 7 3

Ss 3 9

Table 1: Subjective preferences (each cell contains thenumber of subjects with that rating)

DISCUSSION

Even though comparatively little perceptual research hasbeen carried out on the relative strengths of various depthcues, of which only a small portion has addressed issues

specifically related to computer graphic presentation, theresults of our experiment appear to confii some of thoseearlier investigations. In particular, in an early cue conflictstudy, Schriever [10] compared the relative influences ofbinocular disparity (i.e. stereoscopic displays), perspective,shading and occlusion, and showed among other things thedominance of occlusion over disparity information. Morerecently, Bratmstein et at [1] showed that conflicting edge-occlusion dominates disparity. Even more recently,Wickens et al [14] in a review of depth combinationliterature, concluded that motion, disparity and occlusion arethe most powerful depth cues in displays. The resultspresented here clearly contribute to that literature byillustrating some of the powerful advantages that can be

afforded by augmenting visual feedback through eitherpartial occlusion or binocular disparity, and in particularboth in combination.

Two points of particular interest with respect to the silkcursor distinguish this research from other studies. One ofthese is the fact that the silk cursor does not blockcompletely the view of any object which it occludes, due tothe fact that it is semi-transparent. In essence, therefore, wecontend that not only are important enhancements of depthperception to be gained through application of occlusion

cues, but the one clear disadvantage of complete occlusionis greatly diminished – namely, the fact that all information

about objects being obscured by an opaque interveningobject is necessarily lost, For such practical computer-related applications as pursuit tracking, docking, targetacquisition, etc., this is expected to present a significantadvantage.

The second point relates to the fact that the silk cursorprovides discrete, rather than continuous, levels of depth

463

!?!!?HumanFactorsinComputingSystems CM’94* “CekbraliI~g)Medependence”.,_,,_

information. This is in contrast to stereoscopic displays,which are able to provide information not only aboutwherher one object is farther away than another object butalso to a significant extent by how much they are separatedin space, by means of binocular depth scaling. The silkcursor, on the other hand, is not able to provide thisinformation. In Fig. 5 we see that the error rate for the MScase was lower than that of the SW case, which supportsthe effectiveness of the silk cursor as a discrete capturingdevice. However, upon examining Fig. 6 we note that themagnitude of errors for the MS case are larger than those ofthe SW case. The implication of this is that, althoughfewer errors were made with the silk surface cursor, themagnitude of those fewer errors must have been relativelylarger than for the SW case, suggesting that continuousdepth information was not being used,

In summary, we are quite encouraged by these results withthe semi-transparent silk cursor, especially for applicationsin 3D interactive environments. In one existing application– our research on evaluating isometric versus elastic 6 DOF

controllers – we were long hindered by the lack of anadequate display means which would allow us to concentrate

on the control aspects of the experiment, even though wehad already been using a stereoscopic display, the use of thesilk cursor overcame the earlier display bottleneck, andallowed us to conduct that 6 DOF tracking experimentsuccessfully [17].

FUTURE WORK

Of the two novel aspects of the silk volume cursortechnique, this paper has focused on the “silk stocking”effect, i.e. the use of partial occlusion for enhancing depthcues. Our future work will include extending the classicalFitts’ Law model to analyse the “Prince” technique and theeffects of cursor versus target size.

CONCLUSIONWe have proposed a semi-transparent silk volume cursor, toserve as a novel technique for performing target acquisitiontype tasks in 3D environments. Within the context of acarefully designed “virtual fishing” experiment thatrepresented a dynamic 3D target acquisition task, the silkvolume cursor demonstrated superior performance over acomparable wire frame cursor, both in stereo and in monodisplay modes.

ACKNOWLEDGMENTS

We would like to thank the members of the Input ResearchGroup and the Ergonomics in Telerobotics and Control

(ETC) Lab at the University of Toronto, who provided theforum within which this work was undertaken. Primarysupport for this work has come from the InformationTechnology Research Centre of Ontario, the Defence andCivil Institute of Environmental Medicine, the NaturalSciences and Engineering Research Council of Canada, andXerox PARC. Additional support has been provided byDigital Equipment Corp. and Apple Computer Inc. Thissupport is gratefully acknowledged. In addition, the authorswould like to thank Ferdie Poblete for the initial drawing ofthe fish used in the experiment.

REFERENCES1.

2,

3.

4.

5.

6.

7.

8.

9.

10,

11,

12,

13.

14.

15.

16.

17.

Braunstein, M.L., Anderson, G. J., Rouse, M. W.,Tittle, J. S., Recovering viewer-centered depth fromdisparity, occlusion and velocity gradients. Perception& PSydK@ty~iCX 40 (1986). 216-224,Card, S., G. Robertson, and J. Mackinlay. Theinformation visualizer. Proc. of CHI (1991). pp. 181-194.Chen, M., S. J. Mountford, and A. Selten. A study ininteractive 3-D rotation using 2-D control devices.F’roc. o~ACM Siggraph (1988).Evans, K., P. Tanner, and M. Wein. Tablet-BasedValuators That Provide One, Two, or Three Degrees ofFreedom. Computer Graphics. 15(3) (1981). 91-97.Fitts, P.M. The information capacity of the humanmotor system in controlling the amplitude ofmovement. .J. of .Erperimental Psychology. 47 (1954).

381-391.Foley, J. D., A. van Dam, S. K. Feiner, J. F.Hughes. Computer Graphics Principles and Practice.1990, Reading, MA: Addison-Wesley.MacKenzie, I. S., A. Sellen, and W. Buxton. Acomparison of input devices in elementzd pointing anddragging tasks. Proc. of CHI (1991). 161-166.Mackinlay, J. D., S. Card, and G. G. Robertson.Rapid controlled movement through a virtual 3Dworkspace. Computer Graphics, Proc. of SIGGRAPH.24(3) (1990), pp. 197-176.Poulton, E.C., Tracking skill and manual control.1974, New York Academic Press.Schriever, W. Experimentelle Studien iiber dasstereoskopische Sehen. Zeitschriji jilr Pscyhologie. 96(1925). 113-170.SIGGRAPH. Proceedings of Workshop on Interactive3D Graphics. (1986 - 1992). ACM SIGGRAPH.Sollenberger, R, L. and P. Milgram. Effects ofstereoscopic and rotational displays in a three-dimensional path-tracing task Human Factors. 35(3)(1993). 483-499,Venolia, D. Facile 3D direct manipulation. Proc. ofINTERCHI (1993). pp. 31-36.Wickens, C. D., S. Todd, and K. Seedier, Three-dirnensional displays: Perception, implementation andapplications. (1989). CSERIAC Technicil Report 89-001. Wright Patterson Air Force Base, Ohio.Yeh, Y. Y. and L. D. Silverstein. Spatial judgmentswith monoscopic and stereoscopic presentation ofperspective displays. Human Factors. 34(5) (1992).583-600.Zhai, S. and P. Milgram. Human Performance

Evaluation of Manipulation Schemes in VirtualEnvironments. Proc. of VRAIS ’93: IEEE VirtualReality Annual International Symposium. (1993).pp.155-161.Zhai, S. and P. Milgram. Human performanceevaluation of isometric and elastic rate controllers in a6 DOF tracking task. Proc. SPIE Vol. 2057Telemanipulator Technology & Space Telerobotics(1993).

464