Leap motion for sterile manipulation of 3D models

Leapmotionforsterilemanipulationof3DmodelsEuropeanProjectSemesterGroupF

AutumnSemester2017

Supervisor: LouiseOram

Team: GeorgBaumann

TimFink

HubertMarret

JosepMariaPujolMarch

2

3

Abstract

Performingsurgerieswith3Dmodelsaretoanincreasingextentbecomingnormalpractice.

Thecommontoolformanipulatingthese3Dobjectsiswithacomputermouse,whichisnot

designed for a three-dimensional virtual space. Thus, the use of a computer mouse in

combinationwith3Dobjectsisnotintuitive.Thisisthereasonforexperimentingwithnew

inputdevices,suchastheLeapMotiondevice, toenhancetheworkflowofcontrolling3D

objects.Therefore,theprojectdevelopedaprogramformanipulatingthesemodelsinUnity

with the Leap Motion device. A user survey was conducted to identify issues and help

understandhowthesurgeonsuseandinteractdirectlywitha3Dobjectinasterilemanner.

The results of this project strongly support the fact that theusageof the LeapMotion is

moreefficient in various tasks thana computermouse.With this the LeapMotion stacks

uponrevolutionary2Dimaginginmedicalpractice,andthusisanimportantpieceinfuture

medicalcareandsurgery.

4

Acknowledgments

Theprogresswemadewiththeproject“Leapmotionforsterilemanipulationof3Dmodels”

required a great amount of guidance and assistance by students, professors ofHiOA and

staffmembersoftheRikshospitaletUniversityHospital.Wefeltprivilegedtodoresearchon

project,whichisvitalforsurgery.

First,ourdeepestappreciationgoestooursupervisorLouiseOramwhosesupport,guidance

andsolicitudewereinnumerablyvaluablethroughoutthesemester.

Second,wewouldliketothankourprofessorsNinaHagerup,TengelAasSandtrøandPetter

Øyan,fortheirwellorganizedandsupportinglectures.WearealsogratefulofBritBalgaard,

forhercontinuoussupportthroughoutthesemester.

Our gratitude also needs to be expressed to the staff members of the Oslo University

Hospital. Louise Oram established communication between the hospital and the project

team.Inaddition,wefeltprivilegedtoobservetwosurgeries.

WealsothanktheinformationtechnologyconsultancySopraSteriaforlettingusvisitoneof

themostbeautifulworkingplacesinOslo.Besides,theyprovidedusconstructivefeedback

andthepossibilitytotestmixedrealitydevices,suchastheHololens.

Ourprojectteamisverygrateful,thattheHøgskoleniOsloogAkershusmadethisproject

possibleforforeignstudents,byofferingaEuropeanProjectSemester.

Wecannotforgettothankallparticipantswhoparticipatedinourusersurvey.

5

Glossary

SDK ASoftwareDevelopmentKitisasetofsoftwaretools.

3Dobject Amathematicalrepresentationofanysurfaceofanobjectinthree-

dimensions.

collider Aninvisibleframeofa3Dobjectwhichdetectscollisions.

Unity Agameenginefordevelopingvideogamesandsimulationsfor

multipleplatforms.

C# Aobject-orientedprogramminglanguage.

LeapMotion AnAmericancompanywhichmanufacturesasensordevicefor

handmovements.

HiOA OsloandAkershusUniversityCollegeofAppliedSciences

Unix Afamilyofvariousoperatingsystems.

6

7

TableofContents

1. Introduction ___________________________________________________________9

2. ProjectEvaluation______________________________________________________10

2.1. Relevance ______________________________________________________________10

2.2. Effectiveness____________________________________________________________10

2.3. Impact_________________________________________________________________11

2.4. Stakeholderanalysis______________________________________________________12

3. Methods_____________________________________________________________13

3.1. Assessmentoftheproject’sinitialsituation____________________________________13

3.2. Development____________________________________________________________14

3.2.1. Features ___________________________________________________________14

3.2.1.1. Rotation ___________________________________________________________14

3.2.1.2. Scaling_____________________________________________________________14

3.2.1.3. GrabandMove______________________________________________________15

3.2.1.4. ResetButton________________________________________________________15

3.2.1.5. Featurecohesion ____________________________________________________17

3.2.2. Programming _______________________________________________________17

3.2.2.1. WorkingwithUnity3D_____________________________________________________173.2.2.2. GitHubandSourceTree____________________________________________________173.2.2.3. Workflow_______________________________________________________________183.3. Usersurvey _______________________________________________________________18

4. Results_______________________________________________________________34

4.1. Programfunctionality_____________________________________________________34

4.1.1. Rotation______________________________________________________________34

4.1.2. Grabandmove________________________________________________________34

4.1.3. Zoom________________________________________________________________35

4.2. Structureofthecode _____________________________________________________35

4.2.1. Classes_____________________________________________________________36

4.3. Resultsofthesurvey______________________________________________________36

4.3.1. GeneralInformation__________________________________________________37

4.3.2. Results_____________________________________________________________40

8

4.3.2.1. Translation _________________________________________________________41

4.3.2.2. Rotation ___________________________________________________________43

4.3.2.3. Zooming ___________________________________________________________45

4.3.2.4. Button_____________________________________________________________47

4.3.2.5. Additionalstatements ________________________________________________48

5. Discussions___________________________________________________________51

5.1. SurveyAnalysis __________________________________________________________51

5.2. FeatureAnalysis _________________________________________________________52

5.2.1. Grabandmove________________________________________________________52

5.2.2. Rotation______________________________________________________________52

5.2.3. Zoom________________________________________________________________53

5.2.4. Button_______________________________________________________________53

5.3. Suggestions_____________________________________________________________53

5.4. Generalanalysis _________________________________________________________54

6. Conclusion____________________________________________________________56

7. PersonalReflection(Belbin)______________________________________________57

8. References ___________________________________________________________60

9. Illustrationdirectory____________________________________________________61

AppendixA:SourceCode_________________________________________________________63

Loader.cs ___________________________________________________________________63

ButtonManager.cs____________________________________________________________64

ButtonVisibilityManager.cs_____________________________________________________68

Rotation.cs__________________________________________________________________69

Translation.cs________________________________________________________________71

Zoom.cs ____________________________________________________________________74

UIManager.cs________________________________________________________________75

Gesture.cs___________________________________________________________________77

ResetButton.cs_______________________________________________________________78

Singleton.cs _________________________________________________________________79

AppendixC:InstallationManual ___________________________________________________81

AppendixC:MethodsUserStudy __________________________________________________82

ScriptforpresentingtheLeapMotionfeatures _____________________________________82

AppendixD:ResultsUserStudy____________________________________________________83

9

1. Introduction

Leap Motion is a computer hardware device which can sense hand movements. Its

applicationisversatile,yetstillcommonlyusedasaninputdeviceforcomputergames.The

UniversityHospitalofOslorecognizedthepotentialoftheLeapMotionDevicetoimprove

theirworkflowin2013-14andstartedresearchintoit.Theyproposedanideaforaproject

totheOsloandAkershusUniversityCollegeofAppliedSciences.Thus,studentsareableto

strengthentheirskillsinaprofessionalenvironmentandhelpsucceedingtheproject.

The mission of the project “Leap motion for sterile manipulation of 3D models” was to

create a program combinedwith a LeapMotion device, inwhich one canmanipulate 3D

modelsoforganswithouttouchingacomputermouse.

TheLeapMotionDeviceandthegameengineUnity3Dwereusedtodevelopaprogramfor

manipulating3DmodelsinUnity.Theprogramshouldallowtheusertointeractwitha3D

model in various ways. A major goal of the project was to provide cross-platform

compatibilityonmajoroperating systems (Linux,MacOS,andWindows).Themainaimof

thegroupwastodevelopafunctionalprototype,whichonecanfurtherdevelopafterthe

project’stermination.Toensurefurtherdevelopmentoftheproject’sprogram,theproject

groupthoroughlydocumentedthesoftwarecode,includingthethoughtandmotivationof

varioustakenactions.

Additionally,ausersurveywiththefinalprototypewasconducted.Clarificationoftheuser

survey’smethodologyandthediscussionofitsresultsareavailableinthereport.

The report contains a project evaluation, the methodology of the user survey and the

developed program aswell as the results. At the end of this report there is a discussion

about thedevelopedprogramand its implemented features and gestures.Moreover, the

reportincludestheproject’ssustainabilityafterthetermination.

10

2. ProjectEvaluation

2.1. Relevance

Assessingthepresentsituationinwhichsurgeonsperformsurgeryisvitaltoidentifyissues

anddesiresforthisproject.Uponthisassessment,theprojectteamwasabletoformulate

achievablegoalsandspecificobjectives.

Presently surgeons must remain sterile while performing surgery, thus various tools and

objects that they get in contact with have to be sterile too. In that case, operating a

computermouseandakeyboardinordertomanipulate3Dobjectsisdonebyanassistant

oranurse.Basedonapreviousexperiment(Balakrishnan,Retal.,1997),executingtasks,

suchasrotatingormovinga3Dobjectofanorganwithacomputermouseisnotintuitive

andergonomic.Thisbecomesfurtherapparentwhenthesurgeondirectshisorherassistant

tomanipulate the 3D object. Describing a certainmove of a 3D images instead of a 2D

imageappearstobemorecomplex.Hence, it ismore likely tomisunderstandeachother,

resultinginpossibleerrors.

In addition,onemight argue that the current situationofmanipulating3Dobjectswitha

computermouserequirespatienceandtime.Becauseofthat,itiscleartosaythatthefocus

shifts from the patient tomanipulating a 3D object. The efficiency of performing various

tasks in 3D space can be enhanced by utilizing the computermousewith a keyboard, to

executeshortcuts,thusimprovingtheoverallworkflow.Itseemsunlikelythatthisapproach

is used by the majority of surgery assistants or nurses, due to the presumably lack of

knowledge in thesoftwareusedfordisplaying3Dmodels. Inaddition, thegroupbelieves,

the lacking motivation of increasingly utilizing 3D objects prevents it of enhancing the

existing workflow. It is obvious to say, that medical imaging for the human body

revolutionized clinical practice in the past ten to twenty years. This project aims to build

uponpresenttechniques.

2.2. Effectiveness

Evaluating the effectiveness of various approaches and outcomes in correlation to the

projectdesigniskeytounderstandthejustificationofcertainactions.

11

Inthefirstquarteroftheproject'stimeframetheprojectmembersagreeduponimproving

thepriorprojectinUnity.Limitedresourcesanddocumentationrestrictedtheapproachto

enhance the software. Since the software was incompatible with the project’s members

Unix computers, it is safe to say that this approachwould not havemet the initial aims.

Thus, theproject teamdecided to reprogram the software inUnitywithanolder SDK, in

ordertoprovidecross-platformcompatibility.

At this moment, the latest SDK was exclusively on Windows available. Consequently,

compromises had been made by selecting an older software development kit. Various

featuresandfunctionalitywerenotavailableintheSDKversiontheprojectteamchose.This

approach facilitated developing the program and dividing tasks between all four group

members.

This group received supportive feedback from two mentor meetings with professors of

HiOA,inordertoassessandsolveissuesoftheproject.Inaddition,allfourgroupmembers

attended courses in English and Project Management throughout the project cycle. The

project’ssupervisor,LouiseOram,arrangedameetingwiththeproject’scoremembersand

staff members of the Oslo University Hospital Rikshospitalet, in order to see a surgery.

Understanding the environment in which a surgery is being performed was crucial for

designingthesoftware.

2.3. Impact

To ensure the project’s long-term impacts, various objectives have been executed

successfully.Firstly,theprogramwaswrittenintheobject-orientedprogramminglanguage

C#, appropriate for Unity. Secondly, the source code of the software is thoroughly

documented.Lastly,instructionsforinstallingthesoftwarehavebeenmadepubliconlineto

ensureafullyfunctionalsoftwareonmultipleoperatingsystems.

Itiscertaintosaythattheproject’soutcomeswillbenefitsurgeonsandnursesdirectly,due

totheimprovedworkflow.Patientsareconsideredtobeindirectbeneficiaries.

12

2.4. Stakeholderanalysis

The following paragraphs will present the main stakeholders, from core and primary to

secondary.

OuruniqueCore,LouiseOram,worksinthehospitalRikshospitaletasacomputerscientist.

Intheproject,sheactsasanexternalsupervisor.SheproposedtheLeapMotionprojectand

assisted the group by giving continuous constructive feedback and establishing

communicationbetweenRikshospitalet, thecompanySopraSteria,acompanyspecializing

invirtualreality,andtheprojectgroup.

Primary Stakeholder: Rikshospitalet is considered as the project’smain client, due to the

developmentofasoftware,designedforsurgeonsworkingthere.

SecondaryStakeholders:HiOA,theUniversityCollegeprovidesthepossibilitytoenrollinan

European Project Semester. Patients which will be treated by surgeons who use the

project’ssoftwaretoviewthe3Dmodels,arealsoconsideredsecondaryStakeholders.The

EuropeanCommission,whichofferstheEPS,mightalsobeinterestedintheresultsofthis

project. Potential customers, suchasotherhospitals thatwant theproject’s product, can

alsobeconsideredpartofthisgroup.REK,theRegionalCommitteesforMedicalandHealth

Research Ethics, is also considered a secondary stakeholder. Finally, the company Sopra

Steria,whichsupportedtheprojectwithcreativeapproachesandfeedback,isanimportant

stakeholder.

Atthebeginning,theprojectteamdecidedtoconsiderprofessorsasstakeholders,butthey

wereratherconsideredasadvisors,guidingtheworkandgivingfeedback.

Theprimarystakeholdersaremainlyinterestedinutilizingtheprograminthenearfuture.

Theiraimistousetheprogramtoenhancetheworkflowofsurgeries,thussavingvaluable

timeinsurgeriesandmedicalpractice.

The secondary stakeholders are rather keen to see the project’s success. They are not

necessarily concerned about how the program functions, rather the utility of it and the

impacttheprogramhasonitsenvironmentismoreimportantforthem.

13

Theprimarystakeholdersareactiveparticipants,whichcaninfluencetheprojectsdecisions.

Staffmembersof theHiOACollegeonly informtheproject teamingivingthemfeedback,

henceshapingtheoutcomeoftheproject.

3. Methods

3.1. Assessmentoftheproject’sinitialsituation

Thecurrentprojectgroupworkingontheproject“Leapmotionforsterilemanipulationof

3D models” is not the first group working on this project. There have been two groups

before,who participated in the European Project Semester, working on this project. The

members of this team received the results of the groups, which were working on this

project in the previous semesters. The initial idea was to continue working on the

prototypes and results given by the previous project groups. The previous groups were

mainlyworkingonWindows.Sincethedominatingoperatingsysteminthecurrentproject

groupwasMacOS,thecurrentprojectgroupwasnotabletomakethegivenprogramwork

on their computers. To runLeapMotion thecode requiresaSDK (SoftwareDevelopment

Kit)which transforms the signals from LeapMotion into readabledata for the computer.

ThereareseveralSDKs,whichhavebeendevelopedbytheLeapMotioncompanyoverthe

pastyears.Allofthemhavethesamefunctionality,butthenewertheversionisthemore

optimizationsandnewfeatureshavebeenimplementedtotheSDK.ThelatestSDKatthis

time,whichrunsonWindowsandMacOSistheSDK2.3.ThenewestSDK,calledOrion,only

runsonWindows.ThreeofthefourmembersoftheteamownanApplecomputer.Thefirst

ideawas touse theSDKOrion since itwas thenewest SDK,but thiswould require three

teammemberstoinstallWindowsontheircomputersorrunitasavirtualmachine(VM).It

canbemuchslowertorunprogramsinsideaVM,whichwouldhavecausedtheproblem,

thatUnitymightnotworkasfluentasitshould.Afterthat,theteamagreedtousetheSDK

2.3evenifitwasolderthantheSDKOrion.

14

3.2. Development

3.2.1. Features

As thegoalof theprojectwas tobuilda simpleprototype, the teamchose to implement

only the basic functions for 3D model manipulation: Rotating, Translating and Scaling.

However, the Translationmovement has evolved to bemore like aGrab andMove for a

betteruserexperience.

Moreover, another feature had to be implemented to help the user interact with the

application:abuttontoresetthemodeltoitsinitialstate.

3.2.1.1. Rotation

Objectiveofthefeature

The 3D model is static. To see the parts of the organ that are obscured in the current

viewingangle,arotationoftheobjectwillbeneeded.

Conception

Thefirstideawastorotatetheobjectbymovingthehandaroundit,butnottouchingthe

object.The3Dmodelwouldfollowthehand’srotationasitisgrabbedfromadistance.The

main problem of this interaction method is that it would be unclear for the user if the

gestureisactivated.

After reflection, a second idea was to put the hand inside the object and rotate it. The

modelwillmimicthehand’srotation.

Thesecond ideawaseasier to implement,andappearedmore intuitive for theteamso it

wasthegesturechosenforthisfeature.

3.2.1.2. Scaling


The3Dorgancanbereallycomplex.Thus,surgeonsmaywanttozoomonaspecificpartof

themodelforabettervisibility.

15

Conception

Atfirst,theideawasapinchgestureinsidethemodelwithonehand.Thentheusersimply

hastomovethehandpinchingtowardorawayfromtheuser.

Asecondideawasapinchgesturewithbothhandsinsidethemodel.Thenthemovement

wouldbetoseparatethehandspinchingtozoomin,orbringthemclosertozoomout.This

actionwouldscalethe3Dmodel:enlargingittozoomin,shrinkingittozoomout.

Thisconceptprovidesmoreadvantagesthanthefirstone.Themovementwouldmimicthe

zoom gesture on a tablet or smartphone, an action nowadays omnipresent. Besides, the

concept would avoid displacing the viewpoint. Thus, the viewpoint would keep its static

behavior.

3.2.1.3. GrabandMove


At thebeginning,no translationof theobjectwas required.However, ifa zoomhasbeen

performed,thesurgeonsmaywanttolookatanotherpartoftheorganthatisnotvisible.

Theavoidzoomingout,atranslationmightbeexecuted.

Conception

Atfirst,theideawasfistgestureinsidethemodel.Thentheuserwouldonlymovethefistto

dragtheobject.Thedragwouldonlytranslatethemodelwithoutanyrotation.

Nonetheless,thefistgesturewasnotwelldetectedbytheLeapMotion.Athoughtwasto

adopt the pinch gesture used for zooming. The pinch would act as a grab gesture. The

movementthenwouldexecutethesametranslation,butwouldalsobeassociatedwiththe

rotation.Thisfinalgesturewouldrepresentareal-lifebehavior:grabbinganobjectwiththe

handandmovingitaround.

3.2.1.4. ResetButton


16

The3Dobjectwillbedisplacedinthe3Dspaceandthecameraisstatic.Thus,afterseveral

manipulations,theorganmightbedisplacedinthecornersofthescreenorevenoutofit.

The Leap Motion controller does not have a wide range of detection. Therefore it is

complicatedtohavethehandspreciselydetectedwhentheyareatthescreen’sedges.To

avoidthisproblem,onesolutionwastocreateabuttontoresettheobjecttoitsinitialstate.

Itsposition,rotationandscalewouldbereset.

Conception

Toactivateabuttonwithvirtualhands ina3Dapplication isnotacommontask. Indeed,

people nowadays aremostly utilizing their hands to interactwith 2D user interfaces, for

examplesmartphonesandtablets.Therefore,severalapproacheswerestudied.

The Leap Motion Playground applications were tested to study the different ways of

handlingabuttonwithvirtualhands.Inthesecases,thebuttonswere2Dicons.Toactivate

one, the user had to hover his index finger one second on top of it. This solution was

sometimesworkingperfectly.However, itwasoccasionallydifficult toactivate thebutton

withthefingerhoveringoveritwhilenothingwashappening.Hence,anotherapproachhad

tobefound.Thethoughtwastocreatea3Dbuttonandtousethephysicsofthehandto

push it.AnarticlefromFelixNoller(2016),ahumanfactorresearcher,explainedawayto

implementa3DbuttonusingLeapMotionandUnity.Thebuttonispushedbythehandand

is goingback to its original positionwhen thehand leaves it. This last solutionwasmore

ergonomicasonly3Dinteractionwillavailablefortheusers.Theywillnothavetodealwith

both2Dand3Dinteractions.

Figure1:handpushinga3Dbutton

17

3.2.1.5. Featurecohesion

Thefeatureshavebeenconceivedtobeeasilyassociatedwithothers.Therotationcanwork

alone,butisexecutedaswellwhentheusergrabsthemodel.Besides,thezoomfeatureis

usingthepinchgestureofthegrabandmove.

Moreimportantly,thefeatureshavebeenimplementedtobeusedinasamemanipulation,

oneafteranother.Thegoalhereistogivetheuserasensationoffreedommanipulatingthe

organ:as itwasa realobjecthecouldgrab,bringcloser tohiseye, rotateandscalewith

intuitivegestures.Theresetbuttonwouldbepresentheretograntarapidwaytogoback

toastandardpositionofthe3Dobjectandhelptheuseravoidfeelinglost.

3.2.2. Programming

3.2.2.1. WorkingwithUnity3D

Thepreviousgroup thatworkedon theproject chose toassociateUnity3Dwith the Leap

MotionSDK.Unity3D(mostcommonlycalled“Unity”)isapowerful3DEngineusedmostly

for game development but also for any 3D physical interaction software. It has an

ergonomiceditorandacomplexAPI(setofclasses,methodsandfunctionsthatcanbeused

forprogramming).Amultitudeofassetsandpluginsmadebydevelopersarealsoavailable

toaddmorefeaturestotheengine.OneofthemistheLeapMotionplugin,itretrievesthe

LeapMotiondataandgivesittoUnity. Italsoallowsthedevelopertocreatethehandsin

the3Dscenesoitcanplaywiththe3Dmodels.

With all these advantages,Unity has been kept by the team and used for the software

development.

3.2.2.2. GitHubandSourceTree

Whenasoftwareisdeveloped,it isessentialtoutilizeaversioncontroltool.Thiswillsave

severalversionsof thesoftwareonaserver, theymightbeseenasbackups.Therefore, if

oneorvariousfilesarehavingproblems,theycanberestoredtoapreviousversion.

18

Besidesthis,versioncontrolisanefficienttoolforteamworking.Everydeveloperisableto

developonthesameprojectwithouthavingcodeconflictswiththeothers.

Tochoosewhichtoolshouldbeusedfortheapplication,theexperienceddevelopersofthe

team selected Git with the GitHub platform. A web platform used to store and share

projectswithothercoders.Aprojectmanagementtoolispresentinthewebsiteaswell.In

addition, the software SourceTree has been chosen to provide aGraphical Interfacewith

Git.

3.2.2.3. Workflow

Theprojecthasbeenrealized in3phases:Analysis,ProgrammingandSurveying.Foreach

part,thegroupwassplitintosubgroupsorindividualsworkingonsubcomponentstogainin

efficiency.

Firstly,theteamhadtoanalyzetheprojecttofinditsgoals.Theapplicationoftheprevious

grouphasbeenanalyzedtoseewhatcanbedonewithit.Severalresearchesontheinternet

havebeendoneonLeapMotiontoexpandtheteam’sunderstandingonthesubjectandto

look for various approaches. Finally, one teammember learned how to work with Unity

followingafewtutorials.

Secondly,when thegoalsweredefined, the teamstartedcodingon theapplication.Each

memberhadafeaturetoimplement.

Lastly,when a first prototypewas running, a part of the team startedworking on a user

survey.ThegoalwastotesttheapplicationandtostudytheefficiencyofLeapMotion.The

other members continued coding on the application for optimization and to finalize the

documentation.

3.3. Usersurvey

3.3.1. Methodsoftheusersurvey

Aftermostoftheprogramminganddevelopmentpartwasdonetheprojectgrouphadthe

ideatoconductausersurveytoassesspeople’sfirst-timeexperiencewiththeLeapMotion

device.Thereareseveralmotivationstocarryoutausersurvey:

19

1. Thegoalwastosee,iftheimplementedfeaturesareintuitiveorifsomegesturesor

featuresneed tobe reconsidered. It is alwaysnecessary todoearly-stage testingofa

createdproduct,soonecanidentifymistakes,flawsanddesignissues.Timeandmoney

waslimited,sotheprojectgroupwantstogiveaconstructivefeedbacktotheupcoming

groupworkingonthisprojectaboutthecreatedfeatures.Thethoughtbehindthisisto

establishacomfortableprojectstartfortheupcominggroup.

2. By lookingathowpeople interactwithaprototypeofone’sproductonewillgeta

greatamountof information,which isbeneficial forfuturechanges.Conductingauser

survey andUser testing reduces time andmoney investment. By having users testing

one’sproductyouwillgetfeedback.Inthecaseoftheprojectgroup,thefeedbackwas

notonlygivenbyspeakingtothepeople,whoparticipatedintheusersurvey,butalso

byobservinghowtheparticipantsutilizedtheLeapMotiondeviceincombinationwith

the developed program. The information that was collected is important for the

upcominggroupworkingonthisproject.

3. Byconductingausersurvey,onegetsanunbiasedperspectiveofotherpeople.Due

to the biased perspective of the project team members, it was necessary to gather

feedbackandopinions frompeopleoutsidetheproject team.Thereasonfor this is to

spotpossible flawsoftheproduct,whichmightnothavebeenobviousfortheproject

team,duetotheirbiasedperspective.

SigningaConsentFormtoparticipateintheusersurveywasrequired.

TheConsentFormincludedthefollowinginformationandcanbefoundintheappendix.

·Backgroundandpurposeofthesurvey.

·Whatdoesparticipationinthesurveyimply?

·Whatwillhappentotheparticipant’sinformation?

·Aconfirmation,thatitisvoluntarytoparticipateinthesurvey.

In the followingpart theuser surveywill beexplained. Theuser surveywas createdwith

GoogleForms.Inaddition,screenshotsoftheusersurveywillbedisplayed.

20

3.3.2. GeneralInformation

Figure2:GeneralInformationabouttheLeapMotionProject

In the beginning of the user survey the project group briefed each participant about the

LeapMotionprojectandthegoalsofthesurvey.Itneedstobementioned,thatsomeofthe

followingquestions requireananswer,whileothersdonot.Questions,which requiresan

answeraremarkedwitharedstar.At thestartof thesurvey itwasmentioned, thatonly

honestopinionsandanswerstothequestioncanhelptheprojectimprovingtheprogram.

3.3.3. PersonalInformation

21

Figure3:Startofthecollectionofpersonalinformation.Askingaboutthegender.

In this part the participants should click whether they are male or female. It was also

allowedtorefusegivinganinformationaboutyourgenderbyclicking“Abstain”.

Thegoalofthisquestionwastoseeifacertaingenderissignificantlybetterinworkingwith

LeapMotion.

Figure4:Askingabouttheage.

Inthispartofthesurveytheparticipantswereaskedabouttheirage.Thereasonforasking

for age brackets, and not for the specific age of the survey participant,was tomake the

surveymoreanonymous.

Theseagebracketswerechosenforaspecificreason.Thebracket18-25 isconsideredfor

students,whomightbe in theirbachelor’sprogram.Thebracket26-40 ismeant forolder

studentsandpeople,whomighthaveajob.Theoption“Olderthan40”ismeantforpeople,

who did not grow up with current modern technology and therefore may be less

comfortableorfamiliarwithit.Thereasonforthisquestionistosee,whichagebracketcan

handletheLeapMotionprogramthebest.

22

Figure5:Askingabouthowoftenthesurveyparticipantisplayingvideogames.

Heretheparticipantisaskedabouthowfrequentlyoneplaysvideogames.Themotivation

forasking thisquestionwas to see ifpeople,whooftenplayvideogamesare superior in

interactingwiththeLeapMotiondeviceincombinationwiththeproject’sprogram.

Figure6:Askingaboutwhichvideogamestheparticipantplays.

Thereisasimplereasonforaskingthisquestion.Theparticipantmightsaythatsheorheis

playingvideogameseveryday.Ifoneisalreadyexperiencedinplaying3Dgames,heorshe

mightperformbetter,duetoabettersenseoforientationinvirtual3Dspace.Thegoalof

the questions in Fig x+3 and Fig x+4was to find out if oneplays video games andwhich

video game, to see if it would help them interactingwith LeapMotionmore swiftly and

reliable, so the project group could see if they are acting more intuitive with the Leap

Motiondevice.

23

Figure7:Askinginhowmanydimensionsthevideogamesareplayed.

Thisquestionisasimplifiedversionofthepriorquestion.Ifparticipantsareuncertainabout

fillingout thepreviousquestionwitha specificgame title, thisquestionbroadly indicates

whetheroneisfamiliarwitha3Dora2Denvironment.

Figure8:AskingabouthavingexperiencewithLeapMotionorVirtualReality.

There isasimplereasonforaskingabouttheparticipanthavinganyexperiencewithLeap

MotionorVirtualReality.Theprojectgroupexpectedthose,whohavemadeanexperience

withthesethingstointeractbetterandmoreexperiencedwiththeLeapMotiondevice.

Figure9:Askingiftheparticipanthasexperienceinmanipulating3Dmodels.

Thethoughtbehindaskingthisquestionwastoseeifparticipants,whoareexperiencedin

interactingwitha3Dmodelontheircomputerwiththeircomputermouse,wouldalsobe

betterinmanipulatinga3DmodelwiththeLeapMotionDevice.

3.3.4. LeapMotionDemonstrationandexplanationofthevirtualenvironment

Beforeexplaininghowtheusersurveywillbecontinued,thevirtualenvironment,inwhich

theparticipantofthesurveywillwork,needstobeintroducedandexplained.

24

Figure10:Virtualenvironment,inwhichtheparticipantofthesurveyinteracts.

Thisisthevirtualenvironmenteveryparticipantofthesurveywasworkingin.Thefollowing

partwillexplain,whatthenumbersinthepicturearereferringto.

1. Thisisa3Dmodelofadice.Theparticipantofthesurveywillmostlyworkwiththe

diceduringtheusersurvey.Twofeatureswerelinkedtothedice.Itcanbetranslated,

rotatedandscaledbyusingcertaingestures.

2. AHandappearsinthevirtualenvironment,iftheparticipantofthesurveyismoving

hisorherhandabovetheLeapMotiondevice.

3. Abuttonwillappear,iftheparticipantismovinghisorherhandintotherightpartof

the virtual environment. If one’s hand is elsewhere, the button will disappear. The

buttonresetstheposition,therotationandthescaleofthediceoncepushed.

4. “Hva heter du?” is written in a small font under the number six of the dice. This

phrasewillbepartofanupcomingtasktheparticipantsofthesurveyareaskedtosolve.

Thevirtual environmentas it canbe seen in Fig x+9was thedefault virtual environment,

displayedeverytimetheprogramwasstarted.

25

Figure11:IntroductionbeforetheLeapMotionprogramwasdemonstrated.

Each participant followed a procedure. Features, such as translation, rotation, scaling

(zooming) and resetting the dice, were presented. A script of how to demonstrate the

featurescanbefoundintheappendix.

It was important, that no phrases were used, that might push the developed program

towardsapositiveoutcome.

3.3.5. Usertesting

Figure12:Introducingtheupcomingtasks.

InordertoanalyzehowwelltheLeapMotionprogramisworkingfourtaskswerecreated:

·Atranslationtask

·Arotationtask

·Azoomingtask

·Abuttontask

Thegoalwastotesteveryfeature,theprojectgroupdeveloped,indifferentshorttasks.It

needs tobementioned, that the taskswerecounterbalanced.The reason for that is, that

theparticipantsaregainingexperienceabouthowtouseLeapMotionprobablyduringthe

tasks.Ifyouwouldhavethesamelasttaskforeveryparticipant,thetimeparticipantswould

26

needtosolvethistaskwouldalwaysbelowerthanthetimetheywouldneed,iftheywere

doing this task for the first time. TheButtonTaskwas always askedafter the Translation

TaskbecauseitwasrequiredtoknowhowtotranslateforsolvingtheButtonTask.

Thefollowingsequenceswereused:

Tabley:Sequencesusedforthetasks

Rotation Translation Zoom

Translation Zoom Rotation

Zoom Rotation Translation

Foreverytask,thetimewastakenandfilledintoafieldbeneaththedescriptionofthetask.

3.3.6. Translation

Figure13:ExplanationoftheTranslationTask

Thegoalofthetranslationtaskwastopinchwiththefingersandbeinsidethedice,sothe

participantcouldgrabit.Aftergrabbingthedice,thediceshould“touch”thecornersofthe

screeninthesequence,thatwasmentionedinFigx+11.

27

Figure14:movementprocedure

3.3.7. Rotation

Figure15:Explanationoftherotationtask

28

Thistaskaimedtoassesstheunderstandingoftheparticipantbetweenpinchinginthedice

andnotpinching.Bynotpinchinginthediceandjusthavingyourhandinthemodel,the3D

modelofthedicewouldonlyrotatewithyourhandandnotmove.

Figure16:Rotationofthedice

3.3.8. Zooming

Figure17:ExplanationoftheZoomingTask

29

TheZoomingTaskrequiredparticipants tousebothof theirhands inorder tochangethe

scalingofthedice.Thegoalwastoreadthelittlewrittensentence“Hvaheterdu?”.

Figure18:Diceafterchangingitsscaling.

3.3.9. Button

Figure19:ExplanationoftheButtonTask

Forthistaskthepre-knowledgeoftranslatinganobjectina3Dvirtualspacewasrequired.

Thegoalofthistaskwastosee,iftheparticipantscouldunderstandhowdeepthebuttonis

in thevirtual environmentandhowswiftly they couldmoveaway thediceandpress the

buttontwice.

30

Figure20:Pressingthebutton

3.3.10. Feedback

Figure21:Explanationofthereviewingsystem

In the finalpartof thesurvey, theparticipantwereaskedabout theirhonestopinions.As

youcansee inFigx+19, theparticipantswereable toansweronascale from1to5,one

displaying a strong disagreement, while five represents a strong agreement toward the

question.

31

Figure22:FirsttwostatementsabouttheTranslationfeature.

Therewerealwaystwostatementsgiventoonefeature intheLeapMotionprogram.The

firststatementwasaskedwithanegativeadjective.Thesecondstatementwasaskedwitha

positiveadjective. If theparticipantwassatisfiedaboutafeatureandthought itwouldbe

easytouse,sheorhehadtogivea1ora2inthefirststatementanda4ora5instatement

two.Thethoughtbehindthissystemwastonotlettheparticipantfallintoarhythm,where

sheorhewouldalwayssticktothesamenumbers.

Figure23:StatementsabouttheRotationfeature

32

Figure24:StatementsabouttheZoomingfeature

Figure25:StatementsabouttheButtonfeature

The statements about the Button feature differ compared to the prior statements. The

thoughtisstillthesame.Onestatementwithapositiveadjectiveandonestatement,which

isformulatednegative.

ThefollowingquestionsareaboutLeapMotionandhowtheparticipantsfeltusingit.

33

Figure26:StatementsaboutLeapMotioningeneral

Figure27:AdditionalStatementsandendofthesurveybypressingthesubmitbutton.

34

4. Results

4.1. Programfunctionality

Thefinalproductcontainsvariousgesturesandfunctionalitywhichmightdifferfrominitial

objectives.

4.1.1. Rotation

Torotateanobjectinanydirectiononlyonehandisrequired.Thefunctionwillbeexecuted

upon entering the object with one hand. The collider of the object detects if a hand is

touching or entering it and consequently sets a Boolean variable to true. After that, the

rotationofthehandisassignedtotheobject’srotation.Itistobeobservedthatnogesture

isnecessary toactivate the rotation function.Thus,a reliableandcomfortablemethodof

rotatinga3Dobjectisgiven.

4.1.2. Grabandmove

Inorder tomoveanobject,onehas toperformapinchgesture.Nopre-installedgesture

suited thedemandsof a responsive and reliableway to activate the translation function.

Subsequently, the project’s members developed their individual gesture for pinching.

Wheneverthethumbtipisclosertotheotherfingertipsthanaspecificdistance,aBoolean

variableissettotrue.Onlypinchinginsidetheobjectwillexecutethetranslationfunction.

Whilemovingtheobject,onecanstopthisfunctionbyreleasingthepinchgesture.Grabbing

andmovinga3Dobjectwiththistechniquedoesfeelcomfortableandintuitive,duetoits

natural resemblance in human society. Minor issues with the recognition of the pinch

gesturestilloccur,becauseof thephysical limitations theLeapMotioncontroller inherits.

The device has two wide angle infrared cameras, which send the raw data to the

microcontrollerforfurtherprocessing.Ifthepinchgestureiscoveredbythebacksideofthe

palm,thedevicehasnodirect lineofsightofthefingers,thustheLeapMotioncontroller

couldmisinterpretcertainmovements.Forthemostaccurateandreliableexecutionofthe

pinchgesture,thepalmofthehandshouldfacetheLeapMotiondevice.

35

4.1.3. Zoom

Enlarging an object was certainly the most complex function to implement. At first, the

projectgroupimplementedapinchtozoomgesture,whichisalsopresentontouchscreen

devices.Intheory,theobjectshouldenlargeuponenteringitwithonehandandexpanding

thedistancebetweenthethumbandtheindexfinger.Scalingitdown,onehastominimize

thedistancebetweenthesetwofingers.Ifscaleddown,thehandillustratesapinch,which

wouldinterferewiththegrabandmovefunction.Thus,analternativemethodforscalingan

object was implemented. Presently, two hands are required to execute this function.

Pinchingbothhands insidetheobjectandexpandingthespaceofthehandswillscalethe

object in correlation to the distance of the hands. Therefore, amore reliablemethod of

scalinganobjectisgiven.

4.2. Structureofthecode

AmajorpartismanagedbyUnityandtheLeapMotionAPI.Therefore,thestructureofthe

codeisstraightforwardandadaptable.

Figure28:ObjectDiagramoftheApplication

36

4.2.1. Classes

Loader:AttachedtoaGameObject, it is thescript that loads theUIManagerandthus the

Singleton.

Singleton: It is a design pattern. A class that can be instantiated only once and can be

accessedfromeveryotherscriptinsidetheprogram.ItcontainstheUIManager,theorgan

GameObject and its initial state, and a gesture list. This class’s purpose is to contain the

maininformationneededtobeaccessedfromanyotherscript.Furthermore,itisinitializing

thegesturelistandtheorganinitialstate.

UIManager: As it name is suggests, it manages the User Interface. At this moment, it

displaysthefeedbackicons,whichrepresenttheactivatedfeature.

Gesture:Thisclassisaparentclassforeverygesturethatwillbeimplemented.Itcontains

themainspecificationeachgestureshares.

Rotation,Translation,Zoom:Theseclassescontainthegesturesandexecutefeatures.They

areassociatedwiththeorganGameObjectascomponents.

ButtonManager: This class handles the behavior of a button as it is a 3D element in this

application.Itmanagesitsmovementandcallsanactivationmethodifthebuttonispushed.

ResetButton: This class inherits from the ButtonManager to share its behavior. The

Activationmethodcalls theResetOrganmethodof theSingleton to reset theorgan to its

initialstate.

ButtonVisibilityManager: This class is associated with the parent object of the moving

button.Thisparenthasacollidertodetect ifthehandis infrontofthebutton.Whenthe

handentersthecollider,thebuttonappearsanditcanbepushed.Thebuttondisappearsif

thehandexitsthecollider.

4.3. Resultsofthesurvey

Inthispartthefindingsandresultsoftheusersurveywillbepresented.

Intheupcomingpartsthetermaveragewillbeused.Thisvalueiscalculatedlikethis:

37

AverageTime:

𝑥 = 𝑥#

$

#

∗1𝑛

x̅istheaveragetimetakentoperformacertaintaskbyallparticipants.

x_iisthetimetakentoperformacertaintaskbyoneparticipant.

nisthenumberofparticipantstakingpartintheusersurvey.

4.3.1. GeneralInformation

The project group wants to give a brief overview over the profile of the people, who

participatedintheusersurvey.21peopleparticipatedintheusersurvey.

Figure29:Genderdemographicoftheparticipants

7participantsweremale

13participantswerefemale

1participantdidnotwanttogiveinformationaboutitsgender

38

Figure30:Agedemographicoftheparticipants

17participantswerebetween18and25yearsold

4participantswerebetween26and40yearsold

0participantswereolderthan40

Figure31:Frequencyoftheparticipantsplayingvideogames

0participantswereplayingvideogameseveryday

2participantswereplayingvideogames3-4timesaweek

3participantswereplayingvideogamesonceaweek

39

4participantswereplayingvideogamesafewtimesperyear

12participantsarenotplayingvideogamesatall

Figure32:Listofvideogames,thatareplayedbytheparticipants,whoplayvideogames

Figure33:Amountofpeopleplaying2Dor3Dvideogames

Figure34:experiencewithLeapMotionorVirtualReality.

40

19participantshaveneverworkedwithLeapMotionorVirtualReality

2participantshaveworkedwithLeapMotionorVirtualReality

Figure35:experienceinmanipulating3Dobjectsofparticipants.

3participantswereexperiencedinmanipulating3Dmodelsontheircomputer

18participantswerenotexperiencedinmanipulating3Dmodelsontheircomputer

4.3.2. Results

Thefollowingpartwillbeaboutthetakentimetosolveeachtask,theparticipants’opinion

aboutthefeaturesandabriefanalysisonhowataskwassolved.

Before assessing the gathered data, a few abnormalities concerning the execution of the

surveytasksneedtobementioned:

Nearly every participant experienced the LeapMotion for the first time. By having three

minutes of practice, participants could experience the strengths and weaknesses of the

device,butitwasnotenoughtimetoexperiencehowtodealwiththe“designproblems”,

thattheLeapMotiondevicehas.

The detection area of the LeapMotionDevice is limited, therefore itwas hard for some

participantstofindtherightposition,wheretheycouldworkwiththeLeapMotiondevice.

41

Some participants thought one could accelerate the dice by tossing it away. The survey

shouldhaveasked, iftheparticipant is leftorrighthanded,sothepositioningoftheLeap

Motion device could have been adapted. The majority of participants were missing a

physical feedback upon touching the dice or pressing the button. By taking the time, the

participantneededtosolveacertaintask,timepressurewasappliedtoparticipants.

4.3.2.1. Translation

The following figure will show how much time the participants needed to solve the

translationtask.

Figure36:AveragetimestakenfortheTranslationtask

Theparticipantsanswers,tothestatementsaboutthetranslationfeature,areshowninthe

upcomingpart.

42

Figure37:Answerstostatementoneofusersurvey

13outof21peopledisagreedwiththestatement,thatthetranslationfeatureisdifficultto

use.

Figure38:Answerstostatementtwoofusersurvey

18outof21participantsagreedwiththisstatement,whichsuggests,thattheparticipants

weresatisfiedwiththegesture,chosenfortranslatinga3Dobject.

Abnormalitiesduringthetranslationtask

Some did not knowwhat theword “Translation”meant. A few participants lost the dice

because theyweremoving their hand too fast or leaving the detection area of the Leap

43

Motiondevicebytryingtotouchthecorners.Someparticipantsdidnotpinch inthedice,

butmadeafisttotranslateit.Mostofthetimesthetranslationstillworked

4.3.2.2. Rotation

Thefollowingfigurewillshowhowmuchtimetheparticipantsneededtosolvetherotation

task.

Figure39:Averagetimestakenfortherotationtask

Theparticipantsanswers, to the statementsabout the rotation feature, are shown in the

upcomingpart.

44

Figure40:Answerstorotationfeaturestatement

Sevenparticipantsdisagreedwiththestatement,thattherotationfeatureistoodifficultto

use.Eightparticipantsagreedwiththatstatement.Byobservingtheparticipantsperforming

thetask,theprojectgroupfoundout,thatthegesturesfortherotationandthetranslation

featuremightbe too similar. Sinceone can rotate theobjectwith the translation feature

too.

Figure41:Answerstorotationgesturestatement

Although the gesture used for the rotationmight be similar to the gesture used for the

translation,themajorityoftheparticipantsweresatisfiedwiththegesturechosenforthe

rotationfeature.

Abnormalitiesduringtherotationtask

45

Participants pinched in the dice,whatmeans, that they grabbed the dice to rotate. They

werenotsupposedtograbthedice.Theparticipantswerenotinterrupted,andtheycould

solvethetaskinthiswayaswell.Afewparticipantshadtheissue,thatthedicemovedaway

bymistake,whiletheytriedtorotateit.

4.3.2.3. Zooming

Thefollowingfigurewillshowhowmuchtimetheparticipantsneededtosolvethezooming

task.

Figure42:AveragetimestakenfortheZoomingtask

Theparticipantsanswers, to thestatementsabout thezooming feature,areshown in the

upcomingpart.

46

Figure43:Answerstozoomfeaturestatement

Theprojectgroupcouldsee,thatthezoomingfeatureisdifficulttousefornearlyhalfthe

participants.

Figure44:Answerstozoominggesturestatement

Thechosengestureseemstobeacceptedbythemajorityofparticipants.Itisincomparison

totheothergesturestheonlyone,whichrequirestwohands.

AbnormalitiesduringtheZoomingTask

Occasionallythezoomingtaskwassolvedswift,becausethediceglitchedoutandchanged

itsscaletotherightscale,inwhichtheparticipantscouldreadwhatwaswrittenonthedice.

Whenthedice’sscalewasincreasedtoomuch,itfilledthewholescenery.Inthiscaseitwas

hardtodecreasethescaleofthediceagainandsometimestheprogramhadtoberestarted

byonememberoftheprojectgroup.

47

4.3.2.4. Button

Thefollowingfigurewillshowhowmuchtimetheparticipantsneededtosolvethebutton

task.

Figure45:AveragetimestakenfortheZoomingtask

Theparticipantsanswers,tothestatementsaboutthebutton,areshownintheupcoming

part.

Figure46:Answerstobuttoneaseofusestatement

48

Mostoftheparticipantsthought,thatthebuttonwaseasytopress.

Figure47:Answerstoalternativebuttonstatement

Thechosenwayofpushingthebuttonseemsfineaswell.

Abnormalitiesduringthebuttontask

A great number of participants did not have a feeling for the deepness of the scenery.

Therefore,theytriedtopushthebuttoninaplace,whereitwasnot.Peopletriedtopush

thebuttonfromtheside.Thebuttongotstucksometimes.

4.3.2.5. Additionalstatements

Figure48:AnswerstoeaseofuseofLeapMotionstatement

49

Themajorityoftheparticipantswereunsureortendedtoslightlyagree,thatLeapMotionis

easytouse.Thatisunderstandablebecauseitseems,thatLeapMotionincombinationwith

thewrittenprogramrequiressomepracticetomanipulatea3Dmodel.

Figure49:AnswerstoLeapMotionfrustrationstatement

Although Leap Motion requires some experience, most of the participants were not

frustrated by using the LeapMotion. The goal of the surveywas not to show a fun and

innovative technology, although a lot of participants stated, that they had fun doing the

surveyandthatLeapMotionisaninterestingtechnology.

Figure50:AnswerstocomfortableuseofLeapMotionstatement

OnlyonepersonfeltslightlyuncomfortableusingLeapMotion.

50

Figure51:Answerstosupportstatement

15participants said, that theywouldnotneed thehelpof a technician tobeable touse

Leap Motion. That shows the project group, that Leap Motion in combination with the

writtenprogramisintuitiveenoughinitscurrentstate.

Figure52:Answerstosurgerystatement

Nearly every participant mentioned, that this is the statement, which is the hardest to

answer. The reason for that might be, that the participants saw, that the program was

intuitive and goodworking but yet not finished enough to use it during a surgery. Some

peoplethought,thatasurgeonwoulduseLeapMotiontocontrolamachine,whichwould

performsurgeryonthem.Intheendninepeopletendtogoundersurgery,whileasurgeon

51

isusingLeapMotionand8tendtonotgoundersurgerywithasurgeonusingatechnology

likethis.

5. Discussions

5.1. SurveyAnalysis

Theusersurveywasanecessity.ItsaimwastoseehowtheparticipantsinteractwithLeap

Motion in combination with the developed program and their opinions about it. The

gathereddatawasvitalforassessingandimprovingtheprogram.Inretrospect,certainparts

of the user survey should have been changed or rethought. The following paragraph

includesthoughtstheprojectgrouphadduringanalyzingthecollecteddata.

Anumberof21participantsseemssufficient,althoughtheamountshouldhavebeenhigher

insomecases.ThepeoplewhoalreadyhadexperiencewithLeapMotionorexperiencein

manipulationof3Dmodelswerefasterineverytaskthanpeoplewithoutexperienceinany

ofthesefields.Sincethreeparticipantswereexperiencedinmanipulationof3Dmodelsand

two participants had experience with Leap Motion, the project group cannot make a

statisticallyproofedstatement,thatsaysthatthesepeoplearedefinitelyfasterthanpeople

without experience in any of these fields. The project group can state, that people,who

haveexperienceinmanipulationof3DmodelsorexperienceinworkingwithLeapMotion

tendtobefasterthanpeoplewithoutanyexperienceinthesefields.

Since some participants solved taskswithmere luck (glitching out of the dice) and some

participantshad“badluck”tryingtosolvethetasks,itisobsoletetodoastandarddeviation

oftheaveragecalculation.Asolutiontothisissuecouldhavebeentomaketheparticipants

solvethetasksmorethanonetime.Atablewiththestandarddeviationscanstillbefound

intheappendix.Thegroupdecided,thatitismoreusefultousetheMedianinstead.

Thetakentimeoftaskswasdifficulttomeasure. Insomecases,thetimeshouldnothave

beenusedtocalculateanaveragebecauseitwasatimeinfluencedbyluckorbadluck.That

isalsothereason,whycalculatingamedianwasimportant.Theprojectgroupwouldeven

gofurtherandsaythatonecanmakebetterstatementbasedonthemedianthanonthe

average. It was alreadymentioned, that doing a task several times could have fixed this

52

issue. A side effect of this solution would be, that the time of the survey would have

increasedsignificantly.

5.2. FeatureAnalysis

Allprojectmembersdevelopedtheprogram,thustheyhadsufficientpracticewiththeLeap

Motion controller tomanipulate3Dobjects throughout the semester.Most Interviewees,

whoparticipated in theuser survey,werenot familiarusing thiskindof inputdevice ina

virtual3Dspace.Therefore,theprojectteamdiscoveredcriticalissuesoftheprogramfrom

theseinterviewees,whichwerenotobviousfortheteam.

5.2.1. Grabandmove

Accordingtotheusersurvey,themajorityoftheintervieweesfoundthatthegrabandmove

feature (translation)wasnotdifficult touse,sincethegestureused for translating the3D

object was intuitive for more than 85 % interviewees. There were several complaints

concerningthemissingphysicalfeedback,whentouchingthedicebytheparticipants.Given

thiscircumstance,itissafetosay,thatnearlyallintervieweesareaccustomedtoreceiving

physicalfeedbackupontouchingatouchscreenorotherobjects.Fromaclinicalperspective,

thisargumenthas littleornovalidity.Theproject’s targetgroup,whicharesurgeons,can

use the software in combinationwith the LeapMotiondevicedirectlywithnohelpof an

assistantornurse,duetoitssterility.

5.2.2. Rotation

As stated in the results of the user survey, the rotation feature was for at least seven

participantsdifficulttouse.Theprojectteamintentionallydidnottelltheintervieweeshow

toexecutethesefeatures,inordertoseeifitwaspossibletoswiftlylearnthegesturesused

totriggerthefeatures.Thegrabandmovefeaturealsoenablesonetorotateanobject.Due

to this pre-knowledge, several interviewees successfully accomplished the rotation task

withoutusing the rotation feature.Considering this, it isessential toexplain theproject’s

targetgrouphowtoexecutecertainfeaturesandgestures.Thisfeaturerequiresnogesture,

thusitwasforthemajorityoftheparticipantsintuitive.Asreportedbythefindingsofthe

usersurvey,fewintervieweesmovedthedicewhiletryingtoexecutetherotationfeature.

53

Thisisbecausetheirhandswerenotflattenedout,thustriggeringapossibleapinch,which

resultsinactivatingthegrabandmovefeature.

5.2.3. Zoom

Intervieweesat large,seemedtothinkthegestureusedforzoomingwasintuitive,butdid

notagreeupona straightforwardexecutionof the feature.Apossible justification for the

difficulties, when scaling a 3D object in the program, might be the requirement of two

hands.Afewparticipantsdidenlargethedicetoomuch,becausetheirhandswereoutof

theLeapMotiondevicesframe.

5.2.4. Button

Themainconcernaboutthebuttonforseveralintervieweeswasthattheyhadissueswith

perceivingthelocationofthebuttoninavirtual3Dspace.Asreportedintheusersurvey,

participantsdidnothaveafeelingforthedeepnessofthescenery.Otherstriedtopressthe

buttonfromtheside,pushing itnotona linearangle.Consequently, thebuttongotstuck

severaltimes.Toresolveorminimizethisissue,itissuggestedtoanglethebuttontowards

themiddleofthescene,wheretheuserispositioned.

5.3. Suggestions

In this paragraph, there will be various ideas and optimizations explained, that might

improve this project, however they could not be implemented, due to a lack of time. It

seems, that the zoom gesture usually performs accurately while zooming the object.

However,theLeapMotiondoesrarelynotdifferentiatebetweenzoomingandtranslation.

Furthermore, theuseof thepalm to rotateanobject isnot themost intuitivemethod. It

appearedthatsomeuserscouldnotmanipulatetheobjectasfluentlyastheteamhopedto.

Forthisreason,thefollowingimprovementsshouldbecarriedoutwithspecialattentionto

thesefeatures.

Translation: tograb theobjectbypinchingwasanacceptable idea.However,whenusers

turned theirhands faceup, the virtualhandof theprogramsimply couldnotunderstand

that position, because the pinchwill be obscured by the outside of the hand. Thus, it is

recommended to implement a more intuitive gesture. One that mimics grabbing a real

54

object with one hand, by using all five fingers for wrapping around the object and thus,

grabbingtheobject.

Rotation:theuseofthepalmofthehandwaslessintuitive,fewparticipantsthoughttoput

their palm inside the object and then start rotating the object. For this reason, it would

seemlogical tohavethesamehandgesturethetranslationpossesses.Thatwouldreduce

thelearningcurve(timeyouneedtogainnewskills).

Button:attherightpartofthescreenthereisabuttonwhichallowsuserstoresettheinitial

positionof theobject.While therearenosignificantcomplaintsor issueswhenusing this

feature,itwasseenthatwhenlefthanderstriedtopressittheyhadmoredifficultiesthan

thosewhoarerighthanders.So,beforestartingtheprogramthereshouldbeanoptionthat

asksyouwhichyourdominanthandis.

5.4. Generalanalysis

5.4.1. Relevance

Thisprogramhasbeendeveloped inorder to facilitate the interactionof3Dobjects. This

projectcanbringlastingchangesinthemedicalcommunity.Forinstance,partsofthecode

couldperfectlybeusedinotherprogramsorapplications.

5.4.2. Effectiveness

Theobjectiveswerechangedasthesemesterprogressed.Firstly,theobjectivewastoadd

fewadditional features to theprevious code,however theprojectgroupdidnot improve

uponthepriorproject’scode,duetocompatibilityissuesondifferentoperatingsystems.At

thatmoment, objectiveswere drasticallymodified in order to reach the final goal,which

allows surgeons to control three dimensional models while they are working. Another

challenging goal was to develop a program that might be enhanced by future students.

Furthermore, sometimes itwasdifficult toarrangemeetingswith thegroupmembers.At

theend, theproject could achievealmost all themain goals, suchasdevelopaprogram,

which users could manipulate 3D objects this involves translate them, rotate them and

zoomin/outthemandalso,wasabletorunonMacOSandWindows.Theteamrequired

55

additionaltime,inordertocompletetheoptimizationspreviouslymentioned,resultingina

betterinteractionbetweentheusersandtheprogram.

5.4.3. Efficiency

AGanttdiagramwasused tocoordinate tasks throughout thesemester, so thatonegoal

canbedividedinvariousobjectives.Subsequently,objectivesweresplitintotasksforteam

memberstofinishinacertaintimeframe.Tosumup,objectiveswereachievedontime.

5.4.4. Sustainability

Willtherebeanexitstrategyfortheproject?Theremightbeone,astheprogramisworking

wellandmostofthefunctionalitiesareeasytouseevenifoneisnotanadvanceduserin

manipulating3DmodelswithLeapMotion.Exitstrategiesfortheproject:

● Firstly,theprojectmightbeenhancedbythenextEPSstudents.

● Secondly,thisprojectcouldcontributetoimproveopensourceworld,beingusedin

futureapplications,bylettingpeopledownloaditforfree.

● Last one, the program could be sold to a private company as long as they were

interestedinitpreviously.

56

6. Conclusion

In conclusion, a prototype with basic features has been developed with the aid of a

conducted user survey. Additionally, a documentation of the code has been written for

futuredevelopment.

Aftera careful inspectionof thepriorproject’sprogram,and its results, theproject team

agreedupondevelopinganewprogramforthesakeofinstantgesturerecognitionandthe

supportofcross-platformcompatibility.

The group implemented various features, such as rotating, scaling and translating a 3D

object.Additionally,gesturesforexecutingthesefeatureshavebeendeveloped,providing

the possibility to select a feature without physically pressing a button on a keyboard or

computermouse.

Aquantitativemethodwasusedtogetuserdata,andasurveywasconductedamongthe

users.Thegathereddatamadeitpossibletosetupstatisticsandformulatesuggestionsfor

evaluationandassessmentof theprogram.Beside that, theproject teamdiscoveredhow

onemightutilizetheLeapMotiondevicefortheirfirsttimemanipulating3Dobjects.

Intheend,theprojectteamdevelopedaprototypeforsurgeonstomanipulate3Dobjects

directly in amore intuitive and comfortablemanner. Instead of using a computermouse

indirectly,theynowcouldutilizehandgesturestomanipulate3Dobjects.Duetothelackof

time, further research and development could have been made. Hence, the project

publishedthesoftwareonGitHubforfuturedevelopment.LeapMotionhasthepotentialto

enhance the human-computer interaction in a clinical environment. It is capable to

revolutionize medical 3D and 2D imaging, due to its sterility, ease of use and low-cost

barrier.

57

7. PersonalReflection(Belbin)

LeapMotion is a fascinating technology. Aswewere looking through all the projects the

European Project Semester had to offer, wewere all thrilled by the idea of this subject.

Cooperatingwithahospital to improve surgeryusinghigh technologywasanexceptional

opportunityforus.Thisprojectgaveusthechancetoworkonarealproductthatmightlead

toasubstantialimpactforthefutureofsurgery.Therefore,itgrantedusmotivationtostart

theproject.

Although,wetriedtotesttheapplicationoftheformergroupandunderstoodthatitwould

notbepossibletocontinuetheirwork.Achoicehadtobemadebetweentwoversionsof

theLeapMotion.Wedidnotwanttorevertourworkduringthesemester,thusittookusa

longtimetodecidewhichsolutionwillbethemoststable.

Whenourchoiceofstartingfromscratchwiththeolderversionwasmade,wecommenced

programming. The coding levels of the teamwere very heterogeneous. Hence, the tasks

weredivideddifferentlyandtimehadtobegivenfor the leastexperienceddevelopersto

helpthem.Thegoodpointbehindthisisthattheyhavenowabetterunderstandingofthe

developer'swayofworkingandhaveimprovedtheirskillswithprogrammingtoolssuchas

GitHub,UnityandStackOverflow.

Asanalphaversionwasdeveloped,wechosetodividethegroupintwosubgroups.Thetwo

developerskeptcodingontheapplicationforoptimizationwhilethetwoothersweredoing

theusersurvey.Thiswasanexcellent ideabecauseeveryoneworkedmoreefficientlyand

the application was finished in no time. Besides, the user survey gave us a real

understandingof theLeapMotiondeviceandour software.Wealso learneda loton the

methods of surveying and it gave us a first experience on this matter for our upcoming

projects.

Attheend,theprojectachieveditsgoals.Weareproudofwhatwehavedoneandhopeit

willcontinueinthefuturewithanEPSgrouporanyteamofdevelopers.

Thissuccessisnotonlyduetoourselves,butalsotoLouiseOramoursupervisor.Shehelped

uswithvarioustasksandgrantedusconstructivefeedbacksduringtheentireproject.Thus,

58

herexpertiseguidedustowardsaccomplishablegoals.Inaddition,shewasthecontactwith

the hospital and consultant companies that could help us. She gave us a one-time

opportunity to assists a real surgery and invited us to a meeting with Sopra Steria, a

consultancyworkingonvirtualtechnologies.

Inamorepersonalway,our teamhadanoutstandingtimeatHiOA;workingontheLeap

Motionprojectbutmostlyworkingwithexceptionalinternationalstudents.Everyoneinour

group had a different way of working or even seeing the world. Communication was

sometimedifficultandleadtoanargument.However,wealwaysrespectedeachotherand

thedebatealways finishedwithanagreement.Besides,wediscusseda lot tounderstand

eachother,weacceptedeachother’sdifferencesandwetookadvantageofthem.Different

opinionsgivedifferentapproaches,itopenedourfieldofvision.Moreover,aninternational

groupmeansamultitudeofpersonalities.Asyoucanseeinthechartbelow,weweretotally

differentcharacters.Andwealsohaddistinctivebackground,suchasthefieldofstudyor

even the education. We have benefited a lot from this diversity. The most experienced

developerschosetoleadthegroupashewasalsoagoodcoordinatorandshaper.Hence,

he could give the other clear tasks and goals that simplified the workflow dramatically.

Nevertheless,theothermembersofthegrouphaddistinctqualitiesthatimprovedtheteam

work. Some used their technical skills to develop the application. Some used their

communicationskillstorealizetheusersurvey.Finally,someusedtheirwritingskillstohave

aclearacademicreport.

59

Figure53:Belbinresultsofthefourmembersofthegroup

To concludewith, this EPS project taught us several lessons.We learnedwhatwas Leap

Motion and saw that it was not as advanced as it looks like. Some improved their

programmingskillsandtheotherstheirteachingskills.Furthermore,wemainlylearnedhow

to work in an international team, listening to everyone’s opinion, expressing ours and

makingcompromises.Finally,wehadtospeakingEnglish,whenthemomentwasjoyfulor

whenitwasstressful,whenwefeltexcitedorwhenwewereexhausted.Asnomemberof

the team was from an English-speaking country, we all had to do this effort. It was

sometimeschallengingbutweallfaceditandovercameit.Thisprojectwasuniqueandwill

leadustomanyotherinternationalcollaborationsinthefuture.

60

8. References

Balakrishnan,R.,Baudel,T.,Kurtenbach,G.,&Fitzmaurice,G.(1997).TheRockin'Mouse.

ProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems-CHI

'97.doi:10.1145/258549.258778

Noller,F.(2016,July31).Gettingstartedwithsimple3DButtonsusingUnityandLeap

Motion.Retrievedfromhttps://medium.com/@felixnoller/getting-started-with-

simple-3d-buttons-using-unity-and-leap-motion-40d9efded317

61

9. Illustrationdirectory

Figure1:handpushinga3Dbutton____________________________________________16

Figure2:GeneralInformationabouttheLeapMotionProject_______________________20

Figure3:Startofthecollectionofpersonalinformation.Askingaboutthegender.______21

Figure4:Askingabouttheage. _______________________________________________21

Figure5:Askingabouthowoftenthesurveyparticipantisplayingvideogames.________22

Figure6:Askingaboutwhichvideogamestheparticipantplays._____________________22

Figure7:Askinginhowmanydimensionsthevideogamesareplayed.________________23

Figure8:AskingabouthavingexperiencewithLeapMotionorVirtualReality.__________23

Figure9:Askingiftheparticipanthasexperienceinmanipulating3Dmodels.__________23

Figure10:Virtualenvironment,inwhichtheparticipantofthesurveyinteracts.________24

Figure11:IntroductionbeforetheLeapMotionprogramwasdemonstrated. __________25

Figure12:Introducingtheupcomingtasks.______________________________________25

Figure13:ExplanationoftheTranslationTask ___________________________________26

Figure14:movementprocedure______________________________________________27

Figure15:Explanationoftherotationtask______________________________________27

Figure16:Rotationofthedice________________________________________________28

Figure17:ExplanationoftheZoomingTask _____________________________________28

Figure18:Diceafterchangingitsscaling. _______________________________________29

Figure19:ExplanationoftheButtonTask_______________________________________29

Figure20:Pressingthebutton________________________________________________30

Figure21:Explanationofthereviewingsystem __________________________________30

Figure22:FirsttwostatementsabouttheTranslationfeature.______________________31

Figure23:StatementsabouttheRotationfeature ________________________________31

Figure24:StatementsabouttheZoomingfeature________________________________32

Figure25:StatementsabouttheButtonfeature__________________________________32

Figure26:StatementsaboutLeapMotioningeneral______________________________33

Figure27:AdditionalStatementsandendofthesurveybypressingthesubmitbutton.__33

Figure28:ObjectDiagramoftheApplication ____________________________________35

Figure29:Genderdemographicoftheparticipants _______________________________37

62

Figure30:Agedemographicoftheparticipants__________________________________38

Figure31:Frequencyoftheparticipantsplayingvideogames_______________________38

Figure32:Listofvideogames,thatareplayedbytheparticipants,whoplayvideogames 39

Figure33:Amountofpeopleplaying2Dor3Dvideogames_________________________39

Figure34:experiencewithLeapMotionorVirtualReality. _________________________39

Figure35:experienceinmanipulating3Dobjectsofparticipants.____________________40

Figure36:AveragetimestakenfortheTranslationtask____________________________41

Figure37:Answerstostatementoneofusersurvey ______________________________42

Figure38:Answerstostatementtwoofusersurvey ______________________________42

Figure39:Averagetimestakenfortherotationtask ______________________________43

Figure40:Answerstorotationfeaturestatement ________________________________44

Figure41:Answerstorotationgesturestatement ________________________________44

Figure42:AveragetimestakenfortheZoomingtask______________________________45

Figure43:Answerstozoomfeaturestatement___________________________________46

Figure44:Answerstozoominggesturestatement________________________________46

Figure45:AveragetimestakenfortheZoomingtask______________________________47

Figure46:Answerstobuttoneaseofusestatement ______________________________47

Figure47:Answerstoalternativebuttonstatement_______________________________48

Figure48:AnswerstoeaseofuseofLeapMotionstatement _______________________48

Figure49:AnswerstoLeapMotionfrustrationstatement__________________________49

Figure50:AnswerstocomfortableuseofLeapMotionstatement ___________________49

Figure51:Answerstosupportstatement_______________________________________50

Figure52:Answerstosurgerystatement _______________________________________50

Figure53:Belbinresultsofthefourmembersofthegroup_________________________59

63

AppendixA:SourceCode

Loader.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;publicclassLoader:MonoBehaviour{ voidAwake(){ Singleton.Instance.Init(); gameObject.AddComponent<UIManager>(); Singleton.Instance.UIMgr=gameObject.GetComponent<UIManager>(); }}

64

ButtonManager.csusingSystem;usingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingUnityEngine.UI;usingLeap;publicclassButtonManager:MonoBehaviour{ /************** *Attributes* ***************/ publicboollockX; publicboollockY; publicboollockZ; publicfloatreturnSpeed; publicfloatactivationDistance; publicColorinactiveColor; publicColoractiveColor; protectedboolpressed=false; protectedboolactivated=false; publicboolfading=false; protectedVector3startPosition; /*********** *Methods* ************/ //Usethisforinitialization voidStart(){ //Rememberstartpositionofbutton startPosition=transform.localPosition; gameObject.SetActive(false); } //Updateiscalledonceperframe voidUpdate(){

65

floatinitPos=0.0f; if(!lockX)initPos=startPosition.x; elseif(!lockY)initPos=startPosition.y; elseif(!lockZ)initPos=startPosition.z; floatoffset=(activationDistance-initPos)*0.2f; if(transform.localPosition.z>offset) moveButton(); else transform.localPosition=startPosition; if(activated){ activated=false; Activation(); } } protectedvoidmoveButton() { //Uselocalpositioninsteadofglobal,sobuttoncanberotatedinanydirection //Lockdisabledaxis Vector3localPos=transform.localPosition; if(lockX)localPos.x=startPosition.x; if(lockY)localPos.y=startPosition.y; if(lockZ)localPos.z=startPosition.z; transform.localPosition=localPos; //Getdistanceofbuttonpress.Makesuretoonlyhaveonemovingaxis. floatdistance=GetDistance(); //ReturnbuttontostartPosition transform.localPosition=Vector3.Lerp(transform.localPosition,startPosition,Time.deltaTime*returnSpeed); floatpressCompletion=1.0f-((activationDistance-distance)/activationDistance); //buttonpressed if(pressCompletion>=0.7f&&!pressed) { pressed=true; activated=true; //Changecolorofobjecttoactivationcolor StartCoroutine(ChangeButtonColor(gameObject,inactiveColor,activeColor,0.2f)); }

66

//buttonunpressed elseif(pressCompletion<=0.30f&&pressed) { pressed=false; //Changecolorofobjectbacktonormal StartCoroutine(ChangeButtonColor(gameObject,activeColor,inactiveColor,0.3f)); } } protectedIEnumeratorChangeButtonColor(GameObjectobj,Colorfrom,Colorto,floatduration){ floattimeElapsed=0.0f; floatt=0.0f; while(t<1.0f) { timeElapsed+=Time.deltaTime; t=timeElapsed/duration; obj.GetComponent<Renderer>().material.color=Color.Lerp(from,to,t); yieldreturnnull; } } //thebuttonfadesawayanddeactivatesitself protectedIEnumeratorFadeAway(GameObjectobj,floatduration){ fading=true; floattimeElapsed=0.0f; floatt=0.0f; while(t<1.0f) { timeElapsed+=Time.deltaTime; t=timeElapsed/duration; ColorcolorAlpha=obj.GetComponent<Renderer>().material.color; colorAlpha.a=Mathf.Lerp(1.0f,0.0f,t); obj.GetComponent<Renderer>().material.color=colorAlpha; yieldreturnnull; } //Forthefadeawayfunction,deactivatesgameobjectwhentransparent if(gameObject.GetComponent<Renderer>().material.color.a<0.05f){ fading=false;

67

gameObject.SetActive(false); } } //togetthedistancebetweenthebuttonandit'sstartposition protectedfloatGetDistance(){ Vector3allDistances=transform.localPosition-startPosition; floatdist=1.0f; if(!lockX)dist=Math.Abs(allDistances.x); elseif(!lockY)dist=Math.Abs(allDistances.y); elseif(!lockZ)dist=Math.Abs(allDistances.z); returndist; } //actionofthebuttonwhenpressed,fortheButtonManageritischangecolor publicvirtualvoidActivation(){ floatr,g,b; r=UnityEngine.Random.Range(0.0f,1.0f); g=UnityEngine.Random.Range(0.0f,1.0f); b=UnityEngine.Random.Range(0.0f,1.0f); Singleton.Instance.Organ.GetComponent<Renderer>().material.color=newColor(r,g,b); } //thebuttondisappears:fadesaway publicvoidDeactivation(){ if(fading==false){ StartCoroutine(FadeAway(gameObject,0.5f)); } }}

68

ButtonVisibilityManager.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;publicclassButtonVisibilityManager:MonoBehaviour{ protectedGameObjectmovingButton; //Usethisforinitialization voidStart(){ movingButton=transform.Find("MovingButton").gameObject; } //Updateiscalledonceperframe voidUpdate(){ } voidOnTriggerEnter(ColliderotherCollider){ if(otherCollider.transform.parent.name=="middle"&&otherCollider.gameObject.name=="bone1"){ movingButton.SetActive(true); } } voidOnTriggerExit(ColliderotherCollider){ if(otherCollider.transform.parent.name=="middle"&&otherCollider.gameObject.name=="bone1"&&movingButton.activeInHierarchy){ movingButton.GetComponent<ButtonManager>().Deactivation(); } }}

69

Rotation.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingLeap;publicclassRotation:Gesture{ Controllercontroller; publicHandControllerhandCtrl; privateQuaternionoldHandRotation; //Usethisforinitialization voidStart(){ controller=newController(); oldHandRotation=Quaternion.identity; } privateboolIsHand(Colliderother){ if(other.transform.parent&&other.transform.parent.parent&&other.transform.parent.parent.GetComponent<HandModel>()) returntrue; else returnfalse; } voidOnTriggerEnter(){ oldHandRotation=Quaternion.identity; } voidOnTriggerStay(Colliderother){ if(IsHand(other)) { Frameframe=controller.Frame(); Handhand=frame.Hands.Frontmost; QuaternionnewHandRotation=handCtrl.transform.rotation*hand.Basis.Rotation(false); if(oldHandRotation!=Quaternion.identity){ QuaterniondeltaRotation=newHandRotation*Quaternion.Inverse(oldHandRotation); transform.rotation=deltaRotation*transform.rotation; //transform.rotation=handCtrl.transform.rotation*

70

hand.Basis.Rotation(false);//rotation Activated=true; } oldHandRotation=handCtrl.transform.rotation*hand.Basis.Rotation(false); } } voidOnTriggerExit(){ Activated=false; oldHandRotation=Quaternion.identity; }}

71

Translation.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingLeap;publicclassTranslation:Gesture{ Controllercontroller; booltrigger_inside=false; publicbooltrigger_insideL=false; publicbooltrigger_insideR=false; publicHandControllerhandCtrl; publicfloatoldZoomdistance; publicboolisLeftFirst=false; publicboolisLeftSecond=false; publicboolzooming=false; publicbooltranslating=false; publicbooltrigger_pinch1=false; publicbooltrigger_pinch2=false; publicVector3thumb_tip1; publicVector3thumb_tip2; voidStart() { controller=newController(); } voidUpdate(){ Frameframe=controller.Frame(); HandListhandsInFrame=frame.Hands; Handhand=frame.Hands.Frontmost; trigger_pinch1=false; trigger_pinch2=false; Fingerfinger=newFinger(frame.Pointables.Frontmost); floatTHUMB_TRIGGER_DISTANCE=0.025f; HandfirstHand=handsInFrame[0]; HandsecondHand=handsInFrame[1]; thumb_tip1=firstHand.Fingers[0].TipPosition.ToUnityScaled(); thumb_tip2=secondHand.Fingers[0].TipPosition.ToUnityScaled(); isLeftFirst=firstHand.IsLeft?true:false; isLeftSecond=secondHand.IsLeft?true:false; //checkingifhandspinch for(inti=1;i<firstHand.Fingers.Count&&!trigger_pinch1;++i){ Fingerfinger2=firstHand.Fingers[i];

72

for(intj=0;j<22&&!trigger_pinch1;++j){ Vector3joint_position=finger2.JointPosition((Finger.FingerJoint)(j)).ToUnityScaled(); Vector3distance=thumb_tip1-joint_position; if(distance.magnitude<THUMB_TRIGGER_DISTANCE) trigger_pinch1=true; } } for(inti=1;i<secondHand.Fingers.Count&&!trigger_pinch2;++i){ Fingerfinger3=secondHand.Fingers[i]; for(intj=0;j<22&&!trigger_pinch2;++j){ Vector3joint_position2=finger3.JointPosition((Finger.FingerJoint)(j)).ToUnityScaled(); Vector3distance2=thumb_tip2-joint_position2; if(distance2.magnitude<THUMB_TRIGGER_DISTANCE) trigger_pinch2=true; } } //executingthemovement if(handsInFrame.Count>0){ if(finger!=Finger.Invalid){ //translationfunction-ifhandisinsideandpinched->youcanmovetheobject if((trigger_pinch1||trigger_pinch2)&&trigger_inside&&!zooming){ Activated=true; //usedeltaposition transform.position=handCtrl.transform.TransformPoint(hand.Fingers[0].TipPosition.ToUnityScaled());//translation }else{ trigger_inside=false; trigger_insideL=false; trigger_insideR=false; } } } } //checkingifhandsarecollidingwiththeobject voidOnTriggerStay(Colliderother){

73

if(other.transform.parent.root.name=="RigidRoundHandLeft(Clone)"){ trigger_insideL=true; }elseif(other.transform.parent.root.name=="RigidRoundHandRight(Clone)"){ trigger_insideR=true; } trigger_inside=true; } voidOnTriggerExit(Colliderother){ trigger_insideL=false; trigger_insideR=false; Activated=false; translating=false; zooming=false; } }

74

Zoom.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingLeap;publicclassZoom:Gesture{ Controllercontroller; privateTranslationT; voidStart(){ controller=newController(); T=gameObject.GetComponent<Translation>(); } voidUpdate(){ Frameframe=controller.Frame(); HandListhandsInFrame=frame.Hands; booltrigger_insideR=T.trigger_insideR; booltrigger_insideL=T.trigger_insideL; booltrigger_pinch1=T.trigger_pinch1; booltrigger_pinch2=T.trigger_pinch2; boolzooming=T.zooming; floatoldZoomdistance=T.oldZoomdistance; Vector3thumb_tip1=T.thumb_tip1; Vector3thumb_tip2=T.thumb_tip2; if(handsInFrame.Count==2&&trigger_insideR&&trigger_insideL&&trigger_pinch1&&trigger_pinch2){ floatdistance3=Vector3.Distance(thumb_tip1,thumb_tip2)*50; if(oldZoomdistance==distance3){ zooming=false; }else{ zooming=true; Activated=true; transform.localScale=Vector3.one*distance3; } oldZoomdistance=distance3; }else{ Activated=false; } }}

75

UIManager.csusingSystem;usingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingUnityEngine.UI;usingLeap;publicclassUIManager:MonoBehaviour{ publicUnityEngine.UI.ImagegestureImage; //Usethisforinitialization voidStart(){ //InitforGesturefeedbackicon gestureImage=GameObject.Find("GestureIcon").GetComponent<UnityEngine.UI.Image>(); gestureImage.enabled=false; } publicvoidDetectGesture(){ boolimageEnabled=false; foreach(GesturegestureinSingleton.Instance.GetGestureList()){ if(gesture.Activated){ gestureImage.enabled=true; gestureImage.sprite=gesture.Icon; imageEnabled=true; //makingsuretranslatingicongetsselectediftranslatingandrotatingatthesametime. if(gesture.GetType().Equals(typeof(Rotation))){ foreach(GestureotherGestureinSingleton.Instance.GetGestureList()){ if(otherGesture.GetType().Equals(typeof(Translation))&&otherGesture.Activated){ gestureImage.sprite=otherGesture.Icon; break; } } } break; }

76

} if(imageEnabled==false){ gestureImage.enabled=false; } } voidUpdate(){ DetectGesture(); }}

77

Gesture.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingUnityEngine.UI;publicclassGesture:MonoBehaviour{ publicSpriteIcon; publicboolActivated; //Usethisforinitialization voidStart(){ Activated=false; } //Updateiscalledonceperframe voidUpdate(){ }}

78

ResetButton.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;publicclassResetButton:ButtonManager{ //VirtualisationofthefunctionActivation:Resetfunctionisactivated publicoverridevoidActivation(){ Singleton.Instance.ResetOrgan(); }}

79

Singleton.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;publicclassSingleton{ /************** *Attributes* ***************/ privatestaticSingletoninstance; publicUIManagerUIMgr; privateList<Gesture>gestureList; publicGameObjectOrgan; publicVector3initialOrganPosition; publicQuaternioninitialOrganRotation; publicVector3initialOrganLocalScale; /*********** *Methods* ************/ privateSingleton(){} publicstaticSingletonInstance { get { if(instance==null) { instance=newSingleton(); } returninstance; } } publicvoidInit(){ InitOrgan(); InitGestureList(); }

80

publicvoidInitOrgan(){ Organ=GameObject.FindGameObjectWithTag("Organ"); initialOrganPosition=Organ.transform.position; initialOrganRotation=Organ.transform.rotation; initialOrganLocalScale=Organ.transform.localScale; } publicvoidResetOrgan(){ Organ.transform.position=initialOrganPosition; Organ.transform.rotation=initialOrganRotation; Organ.transform.localScale=initialOrganLocalScale; } publicvoidInitGestureList(){ gestureList=newList<Gesture>(); Gesturezoom=Organ.GetComponent<Zoom>(); Gesturerotation=Organ.GetComponent<Rotation>(); Gesturetranslation=Organ.GetComponent<Translation>(); gestureList.Add(zoom); gestureList.Add(rotation); gestureList.Add(translation); } publicList<Gesture>GetGestureList(){ returngestureList; }}

81

AppendixC:InstallationManual

1. DownloadandinstallthelatestversionofUnity(2017.1.1orabove).

2. DownloadtheLeapMotionCoreAssetsv2.3.1fromGitHubandextractit.

3. DownloadtheprojectfromGitHubasaziparchiveandextractthearchive.

(https://github.com/schurlbua/LeapMotion)

4. Open the folder of the project and open the file DefaultScene.unity with Unity,

locatedin/source/LeapUnitySDK2/Assets/DefaultScene.unity.

5. InUnity navigate toAssets -> Import Package -> Custom Package… and open the

LeapMotionCoreAssetsv2.3.1.

6. Onceasmallwindowopens,clickImportandwaittillUnityhassuccessfullyinstalled

theassets.

7. Duetothemissinglinksoftheobject’sscriptsandmaterials,itisadvisedtofollow

furtherinstructionsontheproject’sGitHub.

(https://github.com/schurlbua/LeapMotion)

82

AppendixC:MethodsUserStudy

ScriptforpresentingtheLeapMotionfeatures

“Thereare4 features Iwant to showyoubeforeyouhave3minutes time topracticeon

yourownandexperienceLeapMotion”

Translation:

“NowIwillshowyouhowyoucantranslatethedice.”

Show the translation and emphasize on, that you have to pinch to grab the object. Just

translatethediceabitaround.

Rotation:

“Iwillshowyoutherotationfeaturenow.”

Showtherotation featureandmention, that theprogramdiffersbetweenhavingonly the

handinthecubeandhavingthehandinthecubeandpinching.

Zooming:

“Iwillshowyouthezoomingfunctionnow.”

Showhowyoucanzoominandzoomout.Emphasize,thatyouhavetobewithbothhands

inthecubeandpinch.

Button:

“Iwillshowyoutheuseofthebuttonnow.”

Movethediceawayandpressthebuttonafewtimes.Lateron,thebuttontaskalwayshas

tobeafterthetranslationtask.

GeneralInformation:

Mentionthatthepeopletestingtheprogramcanalwaysaskustoresettheprogram.Ifthey

askthis,wewillresettheprogram,butthetimewillkeeponrunning.

83

AppendixD:ResultsUserStudy

Table1:takentimesofindividualparticipantsinseconds.

Tasks Translation Rotation Zooming

#1 19,53 2,50 10,06

#2 19,88 18,80 15,14

#3 8,96 7,03 1,27

#4 12,72 12,49 70,62

#5 30,79 42,71 27,77

#6 22,62 5,73 11,20

#7 13,97 11,61 8,18

#8 55,36 17,14 8,31

#9 16,80 17,12 71,90

#10 45,30 17,58 8,40

#11 28,99 82,30 12,19

#12 13,76 14,25 11,06

84

#13 10,67 18,93 11,16

#14 36,84 19,05 34,53

#15 26,99 9,84 62,65

#16 14,00 7,82 10,42

#17 12,53 16,49 7,65

#18 14,43 17,58 20,71

#19 54,64 9,40 9,18

#20 39,48 5,21 12,05

#21 28,04 15,83 5,56

Table2:Averagetimesofdifferentcategoriesofparticipants

Task/Rubrik Translation Rotation Zooming Button

Average(21) 25,06 17,59 20,48 21,39

Median(21) 19,88 15,83 11,16 14,66

Boys(7) 19,12 29,14 30,99 19,34

Girls(13) 25,99 12 15,68 23,73

85

18-25(17) 24,43 14,46 19,45 18,63

26-40(4) 27,76 30,89 24,83 35,95

VG:3-4timesa

week(2)42,72 26,06 18,48 14,05

VG:Oncea

week(3)14,42 14,47 12,37 13,16

VG:Afewtimes

peryear(4)16,71 8,75 10,71 15,85

VG:notplaying

any(12)27,57 19,91 26,09 27,46

Leap/VRno(19) 26,22 17,77 20,96 22,67

Leap/VRyes(2) 14,10 15,92 15,89 14,86

3Dno(19) 25,89 18,10 20,75 22,63

3Dyes(2) 20,08 14,55 18,84 17,72

Table3:Standarddeviationsofdifferentcategoriesofparticipants

Task/Rubrik Translation Rotation Zooming Button

Average 13,81 16,56 20,88 16,35

86

Boys 7,35 23,76 26,11 20,04

Girls 13,84 5,55 15,65 14,38

18-25 12,81 8,88 19,18 11,21

26-40 17,17 29,76 26,49 25,04

3-4timesa

week

11,93 16,66 9,30 2,55

Onceaweek 4,46 5,28 8,17 1,09

Afewtimesper

year

4,66 6,18 0,49 5,41

notplayingany 13,71 19,18 25,52 19,61

Leap/VRno 14,03 17,39 21,84 17,02

Leap/VRyes 0,34 1,67 4,83 0,20

3Dno 13,90 17,70 21,84 17,41

3Dyes 12,06 5,35 13,64 5,77

Leap motion for sterile manipulation of 3D models

Documents