-
TobiasHuber,TomWunderling,MarkusPaschold,
HaukeLang,WernerKneist,ChristianHansen
Highly-ImmersiveVirtualRealityLaparoscopySimulation:DevelopmentandFutureAspects
Pre-print version
TobiasHuber,MarkusPaschold,HaukeLang,WernerKneist
DepartmentofGeneral,VisceralandTransplantSurgery,
UniversityMedicineoftheJohannesGutenberg-UniversityMainz,Germany
[email protected]
TomWunderling,ChristianHansen
FacultyofComputerScience,
Otto-von-GuerickeUniversityMagdeburg,Germany
[email protected]
Thisisapre-printofanarticlepublishedintheInternationalJournalofComputerAssisted
RadiologyandSurgery.Thefinalauthenticatedversionisavailableonlineat:
https://doi.org/10.1007/s11548-017-1686-2
-
2
Abstract
PurposeVirtualReality(VR)applicationswithhead-mounteddisplays(HMDs)havehad
animpactoninformationandmultimediatechnologies.Thecurrentworkaimedto
describetheprocessofdevelopingahighly-immersiveVRsimulationforlaparoscopic
surgery.
MethodsWecombinedaVRlaparoscopysimulator(LapSim)andaVRHMDtocreatea
user-friendlyVRsimulationscenario.Continuousclinicalfeedbackwasanessentialaspect
ofthedevelopmentprocess.WecreatedanartificialVR(AVR)scenariobyintegratingthe
simulatorvideooutputwithVRgamecomponentsoffiguresandequipmentinan
operatingroom.Wealsocreatedahighly-immersiveVRsurrounding(IVR)byintegrating
thesimulatorvideooutputwitha360°videoofastandardlaparoscopyscenariointhe
department'soperatingroom.
ResultsClinicalfeedbackledtooptimizationofthevisualization,synchronization,and
resolutionofthevirtualoperatingrooms(inboththeIVRandtheAVR).Preliminarytesting
resultsrevealedthatindividualsexperiencedahighdegreeeofexhilarationandpresence,
withrareeventsofmotionsickness.Thetechnicalperformanceshowednosignificant
differencecomparedtothatachievedwiththestandardLapSim.
ConclusionOurresultsprovidedaproof-of-conceptforthetechnicalfeasibilityofan
customhighlyimmersiveVR-HMDsetup.Futuretechnicalresearchisneededtoimprove
thevisualization,immersion,andcapabilityofinteractingwithinthevirtualscenario.
Keywords:SurgicalTraining,VirtualReality,LaparoscopicSurgery,Human-Computer-
Interaction,Visualization
-
3
1Introduction
VirtualReality (VR) iscurrentlyused insurgical training to
improvepsychomotor
skills that are important for laparoscopic surgery,
suchashand-eye coordination, spatial
orientation,andmanipulationsinthepresenceofafulcrumeffect[1].Nonetheless,theuse
ofVRsimulatorsislimitedindailyclinicalroutine,duetoalackingrealismoftasks,abstract
graphicdesign,andtheawarenessofparticipantstobeinatrainingenvironment[2,3].The
varietyoftaskshasbeenenlargedsincethedevelopmentofVRlaparoscopysimulations,and
currently,abstracttrainingtasksandproceduraloperationsareavailableinVR.Inaddition,
graphic design of simulation tasks and virtual tissue
interaction in VR laparoscopy
simulationhaveimproved.However,usersarecontinuouslyawarethattheyareinatraining
environmentandnotarealsurgicalsituation.Realisticsurroundingsthatincreasetheuser’s
senseofpresenceduringaVRsimulationareonlypossiblewhenperformingteamtraining
sessions, which require more time, infrastructure, and human
resources [4]. Recent
advancementsininformationandmultimediatechnologyhavemadeitpossibletodevelop
VR applications in combination with head mounted displays
(HMDs). These combined
applicationsarecurrentlyusedintheentertainmentindustry,inthemilitary,andinaviation
training. Medical applications include VR-HMD psychological
interventions for
posttraumaticstressdisorders,phobias,cognitiverehabilitationandandpaintreatmentfor
burn victims [5-8].Additionally, VRwill enhance the validity of
clinical, behavioural and
affective and social neurosciences due to more realistic test
scenarios [9]. The rise of
commerciallyavailableVR-HMDsoverthelastyearhasledtothedevelopmentofavirtual
operating room environment to increase the attractiveness and
the degree of presence
duringVRlaparoscopysimulations.Thepresentworkaimedtodescribethedevelopment
processofanimmersiveVRlaparoscopysimulationsetup.Thegoalwastocombineexisting
technologies to create a user-friendly simulation scenario with
high immersion and
presence. Continuous clinical evaluations and feedback from
laparoscopic surgeons
comprisedanessentialpartofthedevelopmentprocess.
-
4
2RelatedWork
ThecurrentuseofVRinmedicalfieldsismainlyduetothevisualpossibilitiesitoffers,
whichareveryhelpfulforprocessessuchaseducation,simulation,planning,navigation,and
even rehabilitation [10]. VR simulations in surgery are used to
teach technical skills,
behavioralskills,andentireprocedurestotraineesandpracticingsurgeonsworldwide[11].
In VR laparoscopy trainers, users perform surgical tasks with
standard laparoscopy
instruments [12]. Studies on VR laparoscopy simulations have
concluded that the skill
acquisition is equivalent to that acquiredwith laparoscopy box
trainer simulations, and
theseskillscanbetransferredtotheoperatingroom(OR)[13,14].Also,abriefpre-surgical
VRwarm-upcanimproveperformanceintheOR[15].
Technical and visual improvements have influenced VR simulators.
Three-
dimensionaldisplayswithpolarizationor shutter
technologieshavebeen integratedand
investigated with differing results [16,17]. On the other hand,
recent developments in
surgery, such as robotic operations, have led to the development
of VR robotic surgery
simulators[18].VisualizationforaVRroboticsurgerysimulationcanbeseenasavariantof
anHMDsimulation,performedontheroboticconsole;however,theORsurroundingshave
not yet been considered in this context. A recent study
described a VR application that
combinedaVRlaparoscopysimulationandalow-immersionHMD,withonlya45°fieldof
view, to produce a virtual scenario, which consisted of a peg
transfer task in a plain,
computer-generatedroom[19].
The latest generation HMDs, which began shipping in 2016,
feature several
technological advancements realized in recent years. High pixel
density displays from
smartphoneswereusedinearlyprototypestoreducethe‘screen-door’effect.Thesedisplays
operatewith a high refresh rate to reduce latency,which
causesmotion sickness.Novel
methods and custom sensors were developed to improve positional
tracking [20].
Asynchronousre-projectionwasintroducedtoreducelatencyevenfurther.Thistechnique
introducessmallchangesontoapreviouslyrenderedframe,accordingtothemostrecent
positional tracking data, which lowers the computational
requirements of the graphics
processingunit(GPU)[21].Othersoftwaresolutionswererequiredtocorrectfordistortion
andchromaticaberrationsthatarisefromthelenses.Thesesolutionswereneededtomap
theHMDtoawiderfieldofviewandcreateamorecomfortablepointoffocus.
-
5
Fig. 1: Custom IVR setupwith LapSimSimulatorwith the4D joysticks
(Simball, G-coder
Systems,Sweden);thegogglesaretheHTCVive(HMD);andtheheadphonesprovidesound.
Aconventionalmonitorisnotneededinthissetup.
3MaterialsandMethods
VirtualRealityLaparoscopySimulationSystem
The basis of our custom setup was a VR laparoscopy simulator
without haptic
feedback(LapSim),purchasedfromSurgicalScienceAB,Goethenburg,Sweden.Itconsisted
ofa27-inchLCDmonitor(AOCInternational,Taiwan),akeyboardandmouse,aWindows7
PC,andSimball™4Djoysticks,withadoublefoot-switch(G-coderSystems).Allhardware
components were mounted on a rolling, height-adjustable array,
and they were readily
accessible,duetotheopendesignofthechassis.ForinteractionswiththeVRenvironment,
the simulator provided Simball 4D joysticks (Fig. 1 andFig. 2).
Their laser-marked ball
joints,withthreedegreesoffreedom,allowedreal-timecalculationsoftheexact3Dangular
position.Theinputdevicesincludedagrasperinstrumentontheleftandrightsides,anda
camera instrument in the center. During our tests, the camera
was not used in any
-
6
laparoscopic task.Thecomputer featuredproprietarysoftware
fromSurgicalScienceAB,
version2015,whichranonWindows7.Thissoftwareallowedtheuser
toperformbasic
trainingtasks,likepegtransferorpatterncutting,butalsocomplex,morerealisticscenarios,
like a cholecystectomy or appendectomy simulation. The software
logged the user's
performancebyrecordingasetoftask-specificparameters.Thisallowedtheassessmentof
execution quality for each performed task; thus, the
individual's improvement in
psychomotorskillscouldbemonitoredovertime.
CustomVirtualRealityHeadMountedDisplaySystem
TwoVR-HMDsolutions fromtheconsumermarketwereconsidered
inouraimto
extendtheexistingLapSimexperiencewithadditionalhardwareandcontent:TheOculus
RiftCV1andtheHTCVive.Comparedtotheconventionalbuilt-inmonitor,HMDsprovidea
widerfieldofview.Withthecombinedheadtrackingandstereoscopicdeptheffects,theuser
isimmersedinanall-aroundvisualexperiencethatismuchclosertorealhumanvisionthan
the2DdisplayoftheLapSim.ThehighrefreshrateandlowlatencyoftheOLEDdisplayson
theHMDwereimportantinachievingminimumsimulatorsickness.WechosetheHTCVive
HMDforthisstudy,becausecomparedtotheOculusRiftCV1, it
featuredaslightlylarger
fieldofviewof110°comparedto101°fortheOculusRiftCV1andalargetrackingareaof
4.6 by 4.6 meters, which allowed highly-immersive, room-scale VR
[22]. A separate
computerwasnecessarytodrivetheHTCViveHMDwithoutinterferingwiththeexisting
LapSimsoftwareandhardware(Fig.2c).AVR-readylaptop(MSIGT72VR-6RE16H51)was
chosenoveracustom-builtdesktopPC forbettermobility.TheMSI
laptopwasequipped
withanIntelCorei7processor(6700HQ,16GBRAM),aNvidiaGTX1070graphicscard(8GB
VRAM),andtheportsnecessaryforconnectingtheHTCVivelinkbox.
VideoandAudioSignalTransfer
Tointegratethevideooutputpreviouslydisplayedonthe2DLapSimmonitor(Fig.
2a) intoavirtualenvironmentcreatedwith
theseparateHTCViveHMDsystem,a frame
grabberwasused(Fig.2d).ThisUSB3.0HDvideocapturedevice(StartechUSB3HDCAP;
Startech, Northampton, UK) received the HDMI output from the
LapSim PC at 1080p
resolutionand60fps,andsent itovertheUSBtothe
laptopviatheHTTPlive-streaming
protocol,HLS.AnHDMIsplitter(Fig.2e)wasinsertedupstreamtoallowthevideosignalto
-
7
beroutedbacktotheLapSimsystemforsimultaneousdisplayonthebuilt-inmonitor.The
simultaneous display allowed other developers to follow the
laparoscopy simulation, as
usual,onthe2Dmonitor.Thus,theotherdeveloperscouldprovidetechnicaladministration
andcontroltheLapSimsoftwarewiththekeyboardandmouse,whiletheuserwaswearing
theHMD. Audio feedback generated by the LapSim software
(e.g.,whenworkingwith a
virtualelectronicdevice)wastransferredviaananalogcablefromtheLapSimPCheadphone
jacktotheMSIlaptopmicrophonejack(Fig.2f).Thefinalaudiomix-downwasperformed
on the laptop. The user wearing the HMD listened to the audio
output with stereo
headphonesconnectedtotheHTCViveHMD.
Fig.2:ComponentdiagramoftheVRlaparoscopysimulationsystemandourcustomizedVR
HMDsystem.a: conventional2Dscreen,b: surgical simulator,
c:VR-readyMSI laptop,d:
HDMItoUSBframegrabber,e:HDMIsignalsplitter,f:stereoaudiocable,g:localnetwork
connection(LAN),h:Simballdataloggerauxiliaryprogram.
-
8
CreationofVirtualRealitySurroundings
The cross-platform game engine, Unity™ (Version 5.4.2), was used
to create the
virtualsurroundingsbyintegratingthevirtualmonitorwiththeSimballjoystickmovements.
Thelivevideostreamfromthescreengrabberwasdisplayedonaplane,whichservedasa
virtualmonitor,with theUnity engine, “WebCamTexture class”. The
original streamwas
rendered at the standard resolution of 1920´1080 pixels. To
optimize performance, the
texturewasdown-sampledto960´540pixelspriortorendering.SoundfromtheLapSim
systemwasintegratedbyplacinganAudioSourceobjectinthescenethatcapturedtheinput
fromthelaptop’smicrophonejack.Theremainingvirtualspacewasfilled,viadraganddrop,
withmodelsofmedicaldevices,furniture,props,andanimatedcharactermodelsofmedical
personnel and the patient. The models could be moved around to
create a setting that
mimickedtheappearanceofanORduringlaparoscopicsurgery.
Analternativewaytocreateavirtualsurroundingwastointegratetheplaybackfrom
a360°video.Tocreatethiseffect,thevideoclipcouldbemappedontoaspherewiththe
MovieTexturefeatureinUnity.TheSteamVRcamerarigwasthenplacedinthecenterofthe
sphereandscaleddowntoaverysmallsizetoadjustforthefixedperspectiveofthe360°
videorecording.
ClinicianFeedback
Four members of the surgical department were continuously
involved in the
developmentprocess.AllfourmembershadpreviousexperiencewiththeVRlaparoscopy
simulator.
TestingPhase
As part of the first clinical pilot investigations, other
members of the surgical
departmentperformedlaparoscopictasksonthesimulator.Hypothetically,adifferencein
performanceduetodistractionsintheimmersivesetupmaybepossible.Furthermore,the
degreeofimmersionintoavirtualworld,theattentiontotheenvironmentandexhilaration
of the participant are important aspects to optimize the
surrounding.We thus used the
validated questionnaire by Nichols et al. to quantify these
aspects [23]. Furthermore,
negativepsychopathologicalaspectsandvegetativesideeffectshavebeendescribedinVR
andhaveleadtothedescriptionofa“codeofconduct”regardVRresearch[24].Thuswe
-
9
investgatedparticipants’heart rate (stress level) andmotion
sicknesswith thevalidated
motionsicknessscalebyKeshavarzetal.[25].
The presented technical data is a comparison of immersive and
regular VR
simulation. This was investigated with the consectuvie
performance of three tasks (peg
transfer,finedissection,andcholecystectomy)afteraninitialwarm-upphaseinregularVR
mode.Theselectedtasksrepresentdifferentaspectsoflaparoscopicsurgery,navigational
maneuver,finepreparationandproceduralaspects.Taskswerealwaysperformedinregular
modefirstandAVRsecondandintheabovementionedorder.Simulatormetricshavebeen
analysedusingthetotalz-scoreofeachtask.Additionally,metricshavebeengroupedinto
categoriestime,handlingeconomicsanderrorsaspreviouslydescribed[15].
Fig. 3: Screenshot of the artificial virtual reality (AVR)
operating room. The virtual
environmentwasmodeled, based on a 3Dmodel kit from Vertigo
Games, Rotterdam. A
virtualinstrumentallowsinteractionwiththeLapSimsimulator.Avirtualmonitor(center)
showsthelaparoscopysimulator'sgraphicoutput.
-
10
4Results
A general overview of the customVR setupwe developed is shown
inFig. 2. All
software and hardware components and connections proved to be
generally technically
feasible.Thesetupwasuserfriendly,easytoassemble,andhighlymobile.
ArtificialVirtualOperatingRoom
We firstdevelopedanartificial virtual reality
(AVR)OR.OurAVRwasbasedona
commerciallyavailablemodular3Dassetkit,whichcontainedafullyequippedsurgicalOR
(VertigoGames,Rotterdam,Netherlands).Itfeaturedanimated3Dcharactersandseveral
modelsofsurgicaldevices,props,andfurniture.Thefeedbackfromtheclinicaldevelopment
teamledtorepositioningcertaincomponentsinthevirtualenvironment.Thepositionofthe
virtualdisplaywasadaptedtocorrespondtothesettinginourclinic.Thepositionsofthe
trocars, sterile equipment, and surrounding teammembers in the
AVRwere changed to
ensurethepositionwascomfortablefortheuserandreflectedtheclinicalsetting(Fig.3).
Additionally,thesizeofthevirtualmonitorwasincreasedtoimprovevisibilityoffinedetails,
but at the same time, it remained realistic. A very small delay
between the joystick
movementsandthecorrespondingmovementsintheVRsurgicalmonitorwasachievedon
the virtual display. Even after the AVR was optimized, clinical
feedback from the
development team revealed that they remained aware of the fact
that they were in an
artificialenvironmenttheentiretime,whichledtoalowdegreeofpresence.Thislimitation
stimulatedtheinitiativetoimprovethesurroundingsandcreateamorerealisticsimulation.
IntegrationofLaparoscopyInstruments
At first, the fourdevelopers remarked that the instruments in
frontof themwere
missingduringtheAVRsimulation.TheSimballinputdevicesprovidedahapticsensationto
the user,whichwas similar to the sensation of holding real
surgical instruments. These
instrumentsweretheonlymeansbywhichtheusercouldphysicallyinteractwiththevirtual
world.Therefore,itwasimportantthatvirtualrepresentationsoftheseinstrumentscould
beseenwhenwearingtheHMD.ThisvisualizationoftheinstrumentsinVRenabledtheuser
tolocateandgrabthedeviceswhilewearingtheHMD.Toachievethiseffect,theinputdata
fromtheSimballshadtobetranslatedintomovementsthatwerethenappliedtothevirtual
-
11
objects represented by the shapes of the simulated instruments.
Because the virtual
environmentwasgeneratedintheVRsystem,andnottheLapSimsystem,theSimballsinput
datahadtobeinterceptedandsentfromonecomputertotheother.Anauxiliaryprogram,
theSimballDataLogger(Fig.2h),whichraninthebackgroundontheLapSimPC,recorded
thedatawithoutinterferingwiththeLapSimsoftware.Eeroutedthedatastreamtopass
throughalocalareanetwork,whichsentittothelaptopthatrantheVRsystem(Fig.3).The
movementsofthevirtualinstrumentswereveryaccurate,accordingtotheusers.Thedouble
foot-switchwasnotvisualizedwithinthevirtualenvironment.However,thisdidnotseem
tobeaproblemfortheclinicianssincethefootswitchisoftennotvisibleintherealoperating
roomaswellduetothesurgicalcovers.
Highly-ImmersiveVirtualOperatingRoom
AnalternativeVRenvironment,thehighlyimmersiveVR(IVR)OR,wascreatedwith
a360°camera(SamsungGear360,SamsungAG,Seoul,Korea).Sphericalvideosequences
wererecordedinsidearealsurgicalORattheUniversityHospitalMainz,Germany.Forthe
video,asurgicalstaffre-enactedthesituationofareallaparoscopicsurgery.Actualmembers
of the departments acted as the patient, scrub nurses,
laparoscopy assistant, and
anesthesiologist (Fig.4).TheORwas first recordedwithout a
scripteddialogue.Later, a
second scenariowas createdwith interactions between the
fictional characters (e.g., the
scrubnurses).Thesecondscenarioincludedsounds,actions,andconversationsthatwere
typicalduringastandardlaparoscopicprocedure.
Thetestingclinicianswerehighlyexhilaratedinresponsetothepresencetheyfeltin
theOR,particularlyduringthesecondscenario.Thesensationofpresencewasevenmore
exhilaratingwhenperformingproceduralsimulationsettings,suchasacholecystectomy,on
the LapSim. Nonetheless, the display resolution was limited in
the AVR and the IVR,
particularlyintaskslikefinedissection,wherebloodvesselsmustbedifferentiated.Despite
manychangesofsettings,thislimitationwasmentionedbyallparticipants.Thatfindingled
totheconclusionthattheresolutionwaslackingwiththeHMD.Down-sizingtheinputsignal
to960´540inUnitydidnotimpactthevisualqualityoftheimagesignificantly,becausethe
virtualmonitoronlycoveredpartoftheuser'sfieldofview.ThenativeresolutionoftheHTC
ViveHMDwasinsufficienttodisplaytheoriginalimageinfullHDresolutionatthatsize.
-
12
Fig.4:Screenshotofthe360°highlyimmersivevirtualreality(IVR)operatingroom(OR).A
sphericalvideowasgeneratedwithaSamsungGear360°cameraintheORattheUniversity
HospitalMainz,Germany.Thevirtualmonitorinthecentershowsthegraphicoutputfrom
thelaparoscopysimulator(LapSim).
TestingPhase
In a previously published pilot study, we found no significant
difference in
performancebetweentheregularVRlaparoscopyandIVR.Theparticipatingsurgicalstaff
washighlyexhilaratedandindicatedahighlevelofpresence[26].Inadifferentapproach,
we previously investigated potential vegetative side effects.
Nausea was present in this
previous investigation in 10% of participants (2/20). These two
female surgeons had a
historyofmotionsickness.AlthoughheartrateswereelevatedduringIVRsimulations,the
elevationswerenotstatisticallysignificant[27].
ThecurrentanalysisoftechnicalperformancecomparedregularVRlaparoscopyand
AVRof16participants.Weobservednostatisticaldifferencesinthetotalz-scoresorinthe
categorizedz-scoresforhandlingeconomics,errors,andtime.Thesetechnicalresultsare
displayedinTable1.
-
13
Table1:Calculatedz-scoresfortasksperformedbyparticipants(n=16)
VRLSession AVRSession Taskparameters Median(IQR) Median(IQR)
P*
Cholecystectomy Total 2.76(-1.18;5.40) 1.07(-2.18;6.14)
0.918Time 0.39(-0.60;0.74) 0.28(-0.86;0.85) 0.836Economics
2.47(-3.00;4.45) 0.93(-0.74;4.72) 0.836Errors 0.80(-1.48;2.33)
0.97(-1.77;2.03) 0.756 FineDissection Total 0.35(-1.58;3.10)
0.39(-3.53;3.35) 0.918Time 0.12(-0.71;0.57) -0.12(-0.59;0.86)
0.959Economics 1.22(-1.19;2.16) 0.12(-1.43;1.63) 1.000Errors
-0.19(-1.57;1.64) -0.18(-1.23;1.49) 0.756 PegTransfer Total
0.59(-1.73;2.06) 1.39(-1.33;2.67) 0.756Time 0.19(-0.62;0.60)
0.28(-0.22;0.72) 0.796Economics 0.23(-0.24;0.79) 0.78(-1.29;1.24)
0.756Errors -0.61(-0.61;1.01) 0.27(-0.60;0.71) 0.679
IQR:interquartilerange;VRL:virtualrealitylaparoscopysimulation;AVR:artificialvirtualrealityoperatingroom*Wilcoxon-Signed-Rank-Test5Discussion
Previously,Bowmanetal.[28]discussedtheimportantaspectsofimmersion,andthe
sensation of presence in VR. These aspects require hardware
components that provide
optimalrefreshrates,framerates,displaysizes,anddisplayresolutions.Inthecurrentstudy,
ourVRsetupincludedanHMDwithpotentiallythebesthardwarecomponentscommercially
available.Nevertheless,
theresolutionintheHMDwaslimitedforveryfinepreparations,
anditmustbeoptimizedtodeterminethesweetspotwiththebestcombinationofasmooth
frame rate and video playback, an acceptable virtual monitor
size, and good image
resolution.Accordingtothe“reality-virtualitycontinuum”[29],thecombinationoftheVR
laparoscopysimulationandavirtualORinthecurrentsetuprepresentedacombinationof
twovirtuallygeneratedworlds.TheVRlaparoscopysimulatorisusuallyperceivedascloser
-
14
toreality thanthenewlydevelopedsetup,becausethesimulation
ispresentedona two-
dimensionalmonitorinanout-of-contextenvironment.Theonlyconnectionlefttorealityis
the haptic user interface (Simball joysticks) which makes the
virtual display of the
instruments evenmore important.However, interaction inAVRand IVR
is currentlynot
possible and should be improved by further technical advances.
An overview of the
limitations in the setupwe developed is given inTable 2,with
corresponding technical
solutionsandratingsofclinicalimportanceandtechnicalviability.Handtracking(e.g.,via
stereocameras)mayincreasespatialawarenessandproprioception,andthisfeaturemay
function as an interface for changing the simulator settings in
VR. Furthermore, several
issuesremainwithHMDtechnologythataredisadvantagestoenhancingtheimmersion,and
intheseaspects,thetraditionalmonitoroutshinesthenewHMDs.Forexample,althoughthe
selectedHMDpossessedhighpixeldensity(above450ppi),atcloseviewingdistances,the
so-calledscreen-dooreffectwhichisdefinedasthevisibilityofinter-pixelspaceswasquite
noticeable [30]. In addition, from an ergonomic standpoint, the
roughly half a kilogram
weight of theHMD can becomeuncomfortablewith prolonged use, and
thisweight also
hindersimmersion.However,newadvancementsmaygiverisetohigher-resolution,lighter-
weight, and possibly wireless HMDs; thus, both the image quality
and weight might be
reduced in future applications to increase user-friendliness.
Furthermore, the cost and
complexitycouldbereducedbycombiningthetwocomputersetupsintoasinglecomputer
system. We demonstrated the technical feasibilities of both the
AVR, produced with a
computer-generatedOR,andtheIVR,producedwiththe360°sphericalvideosequence.
Despitethementionedtechnicallimitations,thecurrentpilotstudy,whichcompared
theAVRtoregularVRlaparoscopy,showedthatparticipantstechnicalperformancewasnot
different toregularVR.Still,anon-randomizedstudydesign isa
limitationregarding the
interpretationofthecurrentresults.CochranereviewscomparingregularboxtrainerstoVR
laparoscopyshowedequivalenceinthosetrainingmethods[14].Thecurrentresultsshow
no significant difference in performance compared to regular VR
surgical simulation
techniques. This was consistent with our previous investigation,
where we compared
regularVRlaparoscopyandIVR.Inthatstudy,althoughwefoundnodifferenceintechnical
performance, the questionnaires revealed that the users
experienced a high degree of
presenceandexhilarationandaratherlowrateofmotionsicknesswiththeIVR[26,27].The
exhilarationexperiencedinVRlaparoscopycombinedwiththeHMDscenarioswasakeyaim
of the current approach, and it is likely to increase the
attractiveness ofVR laparoscopy
-
15
simulation.Furtherstudieswithalargercohortmustbeperformedtoevaluatethegeneral
influenceofAVRandIVRonlaparoscopysimulations.
However,futureresearchshouldplacethegreatestemphasisoninteractionswithin
theVRenvironment,accordingto
theuser’sperformanceontheLapSim.Here, itmaybe
possibletousetechnicaldatafromthesimulatororaspeechrecognitionfeaturetotrigger
differentenvironmentalscenarios,dependingonthesituation.Immersionmaybeimproved
byreplacingstaticvideopartswithhighresolution360°imagesorbyconstructingvirtual
environments with multiple 360° videos. Currently, these
improvements might be
technicallydemanding;however,itwouldnotbedifficulttoimplementbinauralrecordings
ofORsoundsto increasepresenceinthegeneratedworld.Thedegreeof
immersionmay
generallybehigher in thecomputer-generatedAVR than in the
IVR,becauseAVRallows
movements about the virtual room (roomscale VR). However, the
fact that the OR
surroundingsandtheactingindividualswerefamiliartoallparticipantsmayhaveincreased
thedegreeofpresenceintheIVR.Ontheotherhand,inIVR,movementinthevirtualroom
is currently impossible, due to the static video recording.
Movement in IVR might be
achievedby fusingmultiple360° camera recordingsof avirtual
environment to createa
photorealistictypeofAVR.Thelowdegreeofmotionsicknessachievedmightbeexplained
by the fact that all movements were controlled by the
participants, and not by the VR
environment.TheabilitytosimulatecameranavigationwasnotusedinthisfirstVRsetup.
Apotential goalof futuredevelopmentsmightbe to
integrateanassistant thatperforms
camera navigation; however, this integrationmight also
affectmotion sickness. Another
promisingimprovementduetohighly-immersiveVRapplicationcouldbethesimulationof
stress training in the OR as part of the surgeon’s learning
curve. Further research
additionallyneedstofocusonthesimulationofinteractivescenarios.IVRmaybeauseful
tooltosupportthisandinterfacesshouldbedevelopedforsurgicalsimulationsoftwarethat
triggeraspectsinavirtualsurrounding.
Inconclusion,wehavepresentedthetechnicalandclinicaldevelopmentofahighly
immersiveVRlaparoscopysimulationsetup.Thisnewgenerationofsimulationwillenable
clinicalstudiestoevaluatetheimpactofVRforsurgicaltraining.Furthertechnicaladvances
areneededtoimprovevisualizationandinteractivity.Clinicalanalysesshouldfocusonthe
influenceofAVRandIVRonlaparoscopytraining.
-
16
Table2:OverviewofcurrentlimitationsofthecustomimmersiveVirtualRealitySimulationandpossiblesolutionsforfutureresearch,includingratingsoftheirclinicalimportanceandtechnicalviability
Currentlimitation PossiblesolutionClinical
importanceTechnicalviability
Monoaudiorecordingof360°camera
Binauralrecordingofintraoperativesounds
3 3
Missingvisibilityofuser’shandmotion
Handtrackingwithdatagloves 1 2
Missinghaptic/tactilefeedbackfromobjects(patient,table,steriledrapes,otherpersons…)
Useofappropriatehapticfeedbackdevice
3 1
Flatspherewithmissingdepthoffield
Three-dimensional360°cameravideo
3 2
Static360°videoofsurroundings(noroom-scaleVRinIVR)
Multiple360°videostocreateaphotorealisticVRoperatingroomtoenableroom-scaleVR
2 2
NoreactionofsurroundingstoVRsimulatordata(e.g.,mistakes)
RecordingofsimulatordatathatcanbeusedtotriggerdifferentscenariosTextrecognitionofsimulatorcommands
1 3
InteractionwithVRsimulator(administrative)
Connectdataglovestosteersimulationsoftware
1 2
Cameranavigationnotincluded
Performvirtualsurgeriesasateamofsurgeons,andincludeacameranavigator,withtwoVRheadsets
2 1
Simpleoperatingroomscenario
Recorddifferentscenarios(e.g.,stresstraining)
1 3
Surroundingsfamiliartotheparticipants
Recorddifferentoperatingrooms 2 3
Rangeofclinicalimportanceratings:1=highimportance,3=lowimportanceRangeoftechnicalviabilityratings:1=difficult,3=feasible
-
17
Acknowledgments
TheauthorsthankS.Rohde,B.Golla,andM.Kosta(UniversityofMagdeburg)fortechnical
support. Additionally, we appreciate the support from and
discussions with H. Hecht,
DepartmentofGeneralExperimentalPsychology,JohannesGutenbergUniversity,Mainz.
EthicalApproval
Forthistypeofstudy,formalconsentwasnotrequired.Thisstudydidnotincludepatients
oranimals.
Funding
Financial support for the laparoscopy simulatorwas provided by
themedical education
project,“MAICUM”,fromtheMedicalCenteroftheJohannesGutenbergUniversityofMainz.
Fundingfortheadditionalimmersivevirtualrealityhardwarewasprovidedbyintramural
fundingfromtheMedicalCenteroftheJohannesGutenbergUniversityofMainz,Germany,
and educational intramural funding by the Otto-von-Guericke
University Magdeburg,
Germany.
ConflictofInterest
TheauthorsTH,TW,MP,HL,WKandCHdeclarenoconflictsof interestor
financial ties
relatedtothisstudy.
Informedconsent
Thisarticledoesnotcontainpatientdata.
-
18
References
1.
AggarwalR,GrantcharovTP,EriksenJR,BlirupD,KristiansenVB,Funch-JensenP,
DarziA(2006)Anevidence-basedvirtualrealitytrainingprogramfornovice
laparoscopicsurgeons.AnnSurg244(2):310-314.
doi:10.1097/01.sla.0000218094.92650.44
2.
vanDongenKW,vanderWalWA,RinkesIH,SchijvenMP,BroedersIA(2008)Virtual
realitytrainingforendoscopicsurgery:voluntaryorobligatory?SurgEndosc22
(3):664-667.doi:10.1007/s00464-007-9456-9
3.
HuberT,KirschniakA,JohanninkJ(2017)SurveyofTraininginLaparoscopicSkillsin
Germany.ZentralblChir142(1):67-71.doi:10.1055/s-0042-116327
4.
EarleD,BettiD,ScalaE(2017)Developmentofarapidresponseplanfor
intraoperativeemergencies:theCirculate,Scrub,andTechnicalAssistanceTeam.AmJ
Surg213(1):181-186.doi:10.1016/j.amjsurg.2016.01.047
5.
ChiricoA,YadenDB,RivaG,GaggioliA(2016)ThePotentialofVirtualRealityforthe
InvestigationofAwe.FrontPsychol7:1766.doi:10.3389/fpsyg.2016.01766
6.
SchmittYS,HoffmanHG,BloughDK,PattersonDR,JensenMP,SoltaniM,CarrougherGJ,
NakamuraD,ShararSR(2011)Arandomized,controlledtrialofimmersivevirtual
realityanalgesia,duringphysicaltherapyforpediatricburns.Burns37(1):61-68.
doi:10.1016/j.burns.2010.07.007
7.
RizzoA,HartholtA,GrimaniM,LeedsA,LiewerM(2014)Virtualrealityexposure
therapyforcombat-relatedposttraumaticstressdisorder.Computer47(7):31-37
8.
LaverKE,GeorgeS,ThomasS,DeutschJE,CrottyM(2015)Virtualrealityforstroke
rehabilitation.CochraneDatabaseSystRev(2):Cd008349.
doi:10.1002/14651858.CD008349.pub3
9.
ParsonsTD(2015)VirtualRealityforEnhancedEcologicalValidityandExperimental
ControlintheClinical,AffectiveandSocialNeurosciences.Frontiersinhuman
neuroscience9:660.doi:10.3389/fnhum.2015.00660
10.
RivaG(2003)Applicationsofvirtualenvironmentsinmedicine.MethodsInfMed42
(5):524-534
11.
OlaskyJ,SankaranarayananG,SeymourNE,MageeJH,EnquobahrieA,LinMC,
-
19
AggarwalR,BruntLM,SchwaitzbergSD,CaoCG,DeS,JonesDB(2015)Identifying
OpportunitiesforVirtualRealitySimulationinSurgicalEducation:AReviewofthe
ProceedingsfromtheInnovation,Design,andEmergingAlliancesinSurgery(IDEAS)
Conference:VRSurgery.SurgInnov22(5):514-521.doi:10.1177/1553350615583559
12.
PascholdM,SchroderM,KauffDW,GorbauchT,HerzerM,LangH,KneistW(2011)
Virtualrealitylaparoscopy:whichpotentialtraineestartswithahigherproficiency
level?IntJComputAssistRadiolSurg6(5):653-662.doi:10.1007/s11548-010-0542-4
13.
LarsenCR,SoerensenJL,GrantcharovTP,DalsgaardT,SchouenborgL,OttosenC,
SchroederTV,OttesenBS(2009)Effectofvirtualrealitytrainingonlaparoscopic
surgery:randomisedcontrolledtrial.BMJ338:b1802.doi:10.1136/bmj.b1802
14.
NagendranM,GurusamyKS,AggarwalR,LoizidouM,DavidsonBR(2013)Virtual
realitytrainingforsurgicaltraineesinlaparoscopicsurgery.CochraneDatabaseSyst
Rev(8):CD006575.doi:10.1002/14651858.CD006575.pub3
15.
PascholdM,HuberT,KauffDW,BuchheimK,LangH,KneistW(2014)Preconditioning
inlaparoscopicsurgery--resultsofavirtualrealitypilotstudy.LangenbecksArchSurg
399(7):889-895.doi:10.1007/s00423-014-1224-4
16.
KneistW,HuberT,PascholdM,LangH(2016)[3DVirtualRealityLaparoscopic
SimulationinSurgicalEducation-ResultsofaPilotStudy].ZentralblChir141(3):297-
301.doi:10.1055/s-0033-1350609
17.
SorensenSM,SavranMM,KongeL,BjerrumF(2016)Three-dimensionalversustwo-
dimensionalvisioninlaparoscopy:asystematicreview.SurgEndosc30(1):11-23.
doi:10.1007/s00464-015-4189-7
18.
TillouX,CollonS,Martin-FrancoisS,DoerflerA(2016)RoboticSurgerySimulator:
ElementstoBuildaTrainingProgram.JSurgEduc73(5):870-878.
doi:10.1016/j.jsurg.2016.04.008
19.
SankaranarayananG,LiB,ManserK,JonesSB,JonesDB,SchwaitzbergS,CaoCG,DeS
(2016)Faceandconstructvalidationofanextgenerationvirtualreality(Gen2-VR)
surgicalsimulator.SurgEndosc30(3):979-985.doi:10.1007/s00464-015-4278-7
20.
LaValleSM,YershovaA,KatsevM,AntonovMHeadtrackingfortheOculusRift.In:
RoboticsandAutomation(ICRA),2014IEEEInternationalConferenceon,2014.IEEE,
pp187-194
-
20
21.
EvangelakosD,MaraMExtendedTimeWarplatencycompensationforvirtualreality.
In:Proceedingsofthe20thACMSIGGRAPHSymposiumonInteractive3DGraphicsand
Games,2016.ACM,pp193-194
22.
CoburnJQ,FreemanI,SalmonJL(2017)AReviewoftheCapabilitiesofCurrentLow-
CostVirtualRealityTechnologyandItsPotentialtoEnhancetheDesignProcess.
JournalofComputingandInformationScienceinEngineering17(3):031013
23.
NicholsS,HaldaneC,WilsonJR(2000)Measurementofpresenceanditsconsequences
invirtualenvironments.InternationalJournalofHuman-ComputerStudies52(3):471-
491.doi:http://dx.doi.org/10.1006/ijhc.1999.0343
24.
MadaryM,MetzingerTK(2016)RealVirtuality:ACodeofEthicalConduct.
RecommendationsforGoodScientificPracticeandtheConsumersofVR-Technology.
FrontiersinRoboticsandAI3(3).doi:10.3389/frobt.2016.00003
25.
KeshavarzB,HechtH(2011)ValidatinganEfficientMethodtoQuantifyMotion
Sickness.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety53
(4):415-426.doi:10.1177/0018720811403736
26.
HuberT,PascholdM,HansenC,WunderlingT,LangH,KneistW(2017)New
dimensionsinsurgicaltraining:immersivevirtualrealitylaparoscopicsimulation
exhilaratessurgicalstaff.SurgEndosc.doi:10.1007/s00464-017-5500-6
27.
HuberT,PascholdM,HansenC,LangH,KneistW(2017)["Iwilldolaparoscopy
somewhereelse":Total,highlyimmersivevirtualrealitywithoutsideeffects?].Der
Chirurg.doi:10.1007/s00104-017-0465-5
28.
BowmanDA,McMahanRP(2007)VirtualReality:HowMuchImmersionIsEnough?
Computer40(7):36-43.doi:10.1109/MC.2007.257
29.
MilgramP,TakemuraH,UtsumiA,KishinoFAugmentedreality:aclassofdisplayson
thereality-virtualitycontinuum.In,1995.pp282-292
30.
ChoJ-m,KimY-d,JungSH,ShinH,KimTScreenDoorEffectMitigationandIts
QuantitativeEvaluationinVRDisplay.In:SIDSymposium2017.vol1.WileyOnline
Library,pp1154-1156