Skill retention following proficiency-based laparoscopic simulator training

Skill retention followingproficiency-based laparoscopicsimulator trainingDimitrios Stefanidis, MD, PhD, James R. Korndorffer Jr, MD, Rafael Sierra, MD,Cheri Touchard, BS, J. Bruce Dunne, PhD, and Daniel J. Scott, MD, New Orleans, La

Background. Proficiency-based curricula using both virtual reality (VR) and videotrainer (VT)simulators have proven to be efficient and maximally effective, but little is known about the retention ofacquired skills. The purpose of this study was to assess skill retention after completion of a validatedlaparoscopic skills curriculum.Methods. Surgery residents (n = 14) with no previous VR or VT experience were enrolled in anInstitutional Review Board–approved protocol and sequentially practiced 12 Minimally InvasiveSurgical Trainer-VR and 5 VT tasks until proficiency levels were achieved. One VR (manipulatediathermy) and 1 VT (bean drop) tasks were selected for assessment at baseline, after training completion(posttest), and at retention.Results. All residents completed the curriculum. Posttest assessment occurred at 13.2 ± 11.8 days andretention assessment at 7.0 ± 4.0 months. After an early performance decrement at posttest (17%-45%),the acquired skill was maintained up to the end of the follow-up period. For VR, scores were 81.5 ± 23.5at baseline, 33.3 ± 1.8 at proficiency, 48.4 ± 9.2 at posttest, and 48.4 ± 11.8 at retention. For VT,scores were 49.4 ± 12.5 at baseline, 22.0 ± 1.4 at proficiency, 25.6 ± 3.6 at posttest, and 26.4 ± 4.2 atretention. Skill retention was better for VT, compared with VR (P < .02). The extent of skill deteriorationdid not correlate with training duration or resident level.Conclusions. Although residents do not retain all acquired skills (more so for VR than for VT) accordingto simulator assessment, proficiency-based training on simulators results in durable skills. Additionalstudies are warranted to further optimize curriculum design, investigate simulator differences, andestablish training methods that improve skill retention. (Surgery 2005;138:165-70.)

From the Tulane Center for Minimally Invasive Surgery, Tulane University School of Medicine

THE TEACHING of operative skills in the clinicalsetting is constrained by the complexity of proce-dures, medicolegal and ethical concerns, fiscal andtime limitations (especially in the 80-hour work-week), and has created the need for formal train-ing outside the operating room.1-3 Moreover, theincreased incidence of complications that wasobserved as a result of the undisciplined introduc-tion of laparoscopic techniques in the early 1990s4

has raised public awareness and resulted in anoutcry for safety.5 Loss of depth perception6 and

haptic feedback, the fulcrum effect, and the use ofinstruments with limited range of motion7 makelaparoscopic tasks difficult and introduce new skillsets that must be mastered. Acquisition of laparo-scopic skills using both virtual reality (VR) andvideotrainer (VT) simulators can help overcomethe learning curve of new, complex, and difficulttasks, and lead to improved operative perfor-mance.8-12 Moreover, proficiency-based curriculahave proven to be maximally effective and effi-cient.12-14 Besides skill acquisition, skill retention isvitally important15,16 but has not been well studiedfor surgical motor tasks. Contrary to the nonsurgi-cal literature, in which many publications haveinvestigated skill retention,15,17-24 only a handful ofsurgical papers have addressed this issue.25-27 Inaddition, there are no studies that compare thedurability of skill between VR and VT simulators,which have previously demonstrated distinct learn-ing characteristics.28 The purpose of this study wasto investigate the durability of skill that surgery

Presented at the 66th Annual Meeting of the Society ofUniversity Surgeons, Nashville, Tennessee, February 9-12, 2005.

Reprint requests: Daniel J Scott, MD, Associate Professor ofSurgery, Director, Tulane Center for Minimally Invasive Surgery,Department of Surgery, SL-22, 1430 Tulane Ave, New Orleans,LA 70112-2699. E-mail: [email protected].

0039-6060/$ - see front matter

! 2005 Mosby, Inc. All rights reserved.

doi:10.1016/j.surg.2005.06.002

SURGERY 165

residents acquired during a proficiency-based basiclaparoscopic skills curriculum using both VR andVT simulators.

METHODS

Surgery residents (n = 14) of varying levels (R1-4) with no or minimal prior VR or VT experiencewere enrolled in an Institutional Review Board-approved training curriculum on 12 MIST-VR andfive VT tasks. The Minimally Invasive SurgicalTrainer (MIST)-VR (Mentice, Goteborg, Sweden)tasks consisted of 6 core skills 1 (CS1) and 6 coreskills 2 (CS2) tasks on the easy default setting andthe 5 Southwestern VT tasks of the bean drop,running string, checkerboard, block move, andsuture foam drills using a 6-station VT (Karl-StorzEndoscopy, Culver City, Calif), as described indetail elsewhere.29 Scores for the MIST-VR weregenerated automatically by its software on the basisof completion time, errors, economy of move-ment, and economy of diathermy (for tasks usingdiathermy). VT scores were based on completiontime and were recorded with a stopwatch (FischerScientific International Inc, Hampton, NH).

All residents completed a questionnaire regard-ing demographics, handedness, curriculum expec-tations, and prior experience with laparoscopicsurgery, VR or VT simulators, and video games;responses were recorded on Likert scales (Table I).After a standardized demonstration of each taskand administration of a baseline test at the begin-ning of the curriculum (3 repetitions of each task),all residents sequentially practiced the 17 tasksuntil previously established proficiency levels13,14

were achieved on 2 consecutive repetitions. Theproficiency scores were posted at each station sothat trainees could follow easily their own progress.Practice was scheduled during 1-hour weekly ses-sions, and a research assistant was available forassistance although no active instruction was given.

For the evaluation of skill retention, we electedto test trainees on 2 tasks to simplify the testingsessions. We chose the manipulate diathermy(MD) VR task (proficiency score 36.6) and thebean drop (BD) VT task (proficiency score 24) asthese represent the most valid tasks of the 2simulators.12,30

All residents performed 3 repetitions of bothtasks after achieving proficiency (Posttest) and atthe end of the academic year (Retention test).During the follow-up period, no resident hadadditional simulator exposure, but all had routineon-the-job training. Composite scores consisting ofthe mean of the 3 repetitions were calculated andused to compare performance differences. Tocompare relative performance on each simulator,we normalized composite scores according to therespective proficiency levels.

To examine expert performance over time, wetested 3 experts with extensive laparoscopic andsimulator experience on both tasks at a timeremote to any practice on the simulators andwithout allowing any warm up period on the tasks,thus mimicking the testing conditions for thesubjects of this study. One of the experts also hadbeen involved in the creation of the proficiencylevels for both tasks.

Statistical analysis was performed with the use ofa t test, paired t test, and Pearson’s correlation;

Table I. Likert scales and distribution of responses in baseline questionnaire

Laparoscopy self-rating Course expectations1. Very poor 7.7% 1. Useful 69.2%2. Poor 46.1% 2. Somewhat useful 30.8%3. Moderate 38.5% 3. Not useful 0%4. Good 7.7%5. Excellent 0%

Past video game exposure Current video game exposure1. None 7.7% 1. None 69.3%2. Very little 30.7% 2. Very little 23.0%3. Moderate 46.2% 3. Moderate 7.7%4. Extensive 15.4% 4. Extensive 0%

Prior VR experience Prior VT experience1. None 84.6% 1. None 84.6%2. <30 minutes 15.4% 2. <30 minutes 15.4%3. <1 hour 0% 3. <1 hour 0%4. <2 hours 0% 4. <2 hours 0%5. <3 hours 0% 5. <3 hours 0%

SurgeryAugust 2005

166 Stefanidis et al

P less than .05 was considered significant (SPSSSigma Stat, Chicago, Ill). Values are expressed asmean ± SD, unless noted otherwise.

RESULTS

All 14 residents successfully completed the cur-riculum; six R1, four R2, two R3, and two R4residents participated. Mean age was 29.9 ± 3.1years; 12 residents were male and 2, female; 12residents were right handed, one, left handed, andone used both hands equally. Questionnaire dataare shown in Table I.

Training on the MD and the BD tasks required22.3 ± 15.3 and 19.9 ± 9.1 (P = NS) repetitions,respectively. The interval between achieving profi-ciency and posttest was 13.2 ± 11.8 days, andbetween proficiency and retention, 7.0 ± 4.0months.

As seen in Figure 1, after an early performancedecrement at posttest, there was no skill loss forboth tasks at retention. In particular, for MD,scores were 81.5 ± 23.5 at baseline, 33.3 ± 1.8 atproficiency (59% improvement, compared withbaseline, P < .001), 48.4 ± 9.2 at posttest (55%skill retention, compared with proficiency, P <.001), and 48.4 ± 11.8 at retention (0% deteriora-tion, compared with posttest, P = NS). For BD,scores were 49.4 ± 12.5 at baseline, 22.0 ± 1.4 atproficiency (56% improvement, compared withbaseline, P < .001), 25.6 ± 3.6 at posttest (83%skill retention, compared with proficiency, P <.001) and 26.4 ± 4.2 at retention (3% deteriora-

tion, compared with posttest, P = NS). The earlyperformance decay was greater for MD, comparedwith BD (45% vs 17%; P < .001). As seen in Figure2, most of the skill loss occurred within the first 2weeks after training and stabilized thereafter forthe duration of follow-up. In contrast, as expected,the mean performance scores of the 3 experts onboth tasks at a time remote to any simulatorpractice (37.0 for MD and 21 for BD) was veryclose to our proficiency levels (36.6 for MD and 24for BD). Since only 1 of the 3 experts had beeninvolved originally in the creation of our profi-ciency levels, we specifically examined his perfor-mance and found no changes over time (35.9 ± 2.1at retest vs 36.6 ± 5.2 at level development for MDand 24.6 ± 3.2 vs 22.6 ± 5.4 for BD, respectively;P = NS for both).

There was no correlation of skill loss withresident level, duration of training (number ofrepetitions), or any of the other demographic andquestionnaire parameters. Resident level and pastlaparoscopic experience correlated with baselineperformance (indicating construct validity) butonly for the BD (r = 0.68-0.71, P < .01). As seenin Figure 3, there was no difference in the perfor-mance between junior and senior residents for theMD task at any time interval: The BD baselineperformance of seniors was superior to that ofjuniors (indicating further construct validity).

Fig 1. Comparison of trainees’ performance on thebean drop and manipulate diathermy tasks. Scoreshave been normalized to proficiency levels (set at100%), and high scores represent inferior performance.P values reflect differences between the 2 tasks. For bothtasks, performance scores are different for all testing ses-sion comparisons (P < .01) except between Posttest andRetention. Error bars represent SEM.

Fig 2. Skill loss over the follow-up period (7 ± 4months), that has been broken down into 4 intervals.After an initial loss within the first 2 weeks post training,trainee performance remained stable for the duration ofthe follow-up. P values refer to differences between the 2tasks at each time frame. Both Posttest and Retentionperformances of each resident are included in thisgraph. Error bars represent SEM.

SurgeryVolume 138, Number 2

Stefanidis et al 167

DISCUSSION

As seen in previous studies, our traineesachieved significant improvement in performanceafter completing a carefully structured proficiency-based curriculum on previously validated VR andVT simulators; skill acquisition was similar for bothsystems (59% and 56% improvement, comparedwith baseline, respectively). Despite an early per-formance decrement (45% for VR and 17% for VTat 13.2 ± 11.8 days post training) the acquired skillpersisted over a mean 7-month follow-up period.While the early skill loss may seem substantial, littleis known regarding skill retention after simulator-based surgical training. Grober et al25 showed thatretention of skills was better at 4 months afterhands-on bench model training for rodent vasdeferens anastamoses, compared with didactictraining alone. Torkington et al26 showed deterio-ration in basic laparoscopic skill according toMIST-VR testing using the manipulate diathermy

task (25% over 3 months); however, training wasconducted on a box trainer, not on the VR system.Anastakis et al27 found that a single session of asurgical skills curriculum did not confer long-termbenefit to participants. Nonetheless, many morestudies from nonsurgical fields have been pub-lished on durability of skill.15,17,18,21-23 Factorsconsidered important for skill retention includethe duration of retention interval, the quality andquantity of the original training, certain task char-acteristics, and individual differences.15

While a meta-analysis of skill retention foundthat performance decay increased with longerretention intervals and documented a 92% skillloss at 1 year,15 this finding appears to be moreapplicable to cognitive tasks because several au-thors have shown long-term persistence of psycho-motor skills.22,23 Likewise, in our study theretention interval did not appear to be related toskill deterioration because we found that skillacquired during a proficiency-based training per-sisted for many months despite an early decrementin the first few weeks post training.

The quality and quantity of training are alsoimportant factors for skill retention.15 Eventhough the quantity of training (number of repe-titions) did not appear to be related to skillretention, the quality of the training in this study(proficiency based) may be accountable for thelongevity of the acquired skill; having to achieveexpert-derived performance may confer skills thatare resistant to decay. On the other hand, it isunrealistic to believe that the trainee’s ability toachieve the expert-derived performance goals in2 consecutive attempts (as was the endpoint oftraining for our subjects) provides them withenough skill to perform consistently at an expertlevel. Indeed, we showed in this study that whileexpert performance, as measured by simulators,remains stable and consistent over time, traineeslost 20% to 45% of the skill they achieved duringproficiency-based training. Along those lines, theperformance achieved during the posttest actuallymay reflect true trainee ability more accurately,compared with the performance achieved at theend of intensive training. Thus, even though weused a proficiency-based curriculum, it may nothave been rigorous enough or ideal for maximalskill retention; alternative protocols for initialtraining or ongoing practice may be needed tomaintain expert level performance.

Individual differences like baseline ability canalso affect learning, as higher skilled individualsacquire and retain more skill over time.15,17,19

However, in our study, skill retention was not

Fig 3. Resident performance is shown according tojunior (R1-2) and senior (R3-4) levels. There were nostatistically significant differences between levels exceptfor baseline performance on the bean drop (P < .05).Error bars represent SEM.

SurgeryAugust 2005

168 Stefanidis et al

influenced by resident level even though seniorresidents outperformed junior residents at base-line on both tasks (indicating construct validity;statistically significant only for VT). This lack ofcorrelation could be a consequence of our profi-ciency-based curriculum: All residents trained tothe same expert-derived level, and their perfor-mance at training completion was by far superiorto their baseline. In this context, acquired skill(quality of training) appears to be the primarydeterminant of retention, and baseline skill (resi-dent level) may be of less importance, especiallyfor basic laparoscopic skills.

Another factor, motivation, also plays an impor-tant role in skill acquisition31 and may influencelong-term retention.15 Our baseline questionnairedata revealed that all participants were highlymotivated because 100% expected to benefitfrom training. Motivation during training, how-ever, may be affected by the type of task. Inpsychologic terms, ‘‘natural tasks are generallyretained better than artificial tasks’’ becausetrainees exhibit a higher level of interest in learn-ing tasks that seem ‘‘natural’’.15 A study by Ham-ilton et al28 showed a clear preference of traineesfor VT over VR, citing better visualization andtactile feedback that made VT more realistic. Inregards to the significantly higher early skill decayfor the VR system, the superior face validity (real-ism) of the VT task may have motivated thetrainees to maximize their learning and accountedfor the lower skill deterioration on that task. Inaddition, interface differences between the 2 sys-tems may have contributed to the skill retentiondifferences at posttest. Although the VR traininghas proven effective in enhancing operative per-formance and incorporates sophisticated met-rics,12 the environment is artificial and relativelygamelike, compared with VT. While some of the‘‘tricks’’ for obtaining a good score easily may beforgotten for both VR and VT (possibly explainingthe early performance decay), the VR systemseemed to have many more such tricks and wasmuch less forgiving as judged by the higher skillloss at posttest (45% vs 17%, respectively). Never-theless, performance, as measured by both simu-lators, remained stable over a long period of time,substantiating their value as training tools. Thisknowledge should be taken into considerationwhen simulators are used as training or testingtools. Undoubtedly, as technology continues toimprove, so too will the fidelity of the simulators,especially for VR.

While the findings of this study may not begeneralized easily to the clinical setting, the results

have important implications for training curriculadesign for residents and continuing education forpracticing surgeons. The knowledge of how muchskill is retained and when deterioration occurs (ifit does) allows educators to plan retraining ses-sions at appropriate time frames to maintain opti-mal performance. Moreover, applying automaticitytheory in training (how experts become auto-mated at performing a task) and defining profi-ciency in new multidimensional fashions mayprove more effective. These issues may be of lesserimportance when, after initial training, a surgeonhas ongoing clinical practice on a task that closelyresembles the simulator task (eg, laparoscopicsuturing); in that instance the surgeon’s skill willbe unlikely to deteriorate because of ongoingpractice. However, skill decay occurs potentiallywhen a surgeon neither performs a procedure noruses a specific skill for a long time; refreshertraining on a simulator, such as just before theprocedure, might be of great value. In addition,although simulator training has proven to improveoperative performance,8-12 the longevity of thatbenefit is unknown. Such studies are currentlyunderway in our laboratory.

CONCLUSION

This study clearly shows long-term benefit fortrainees after proficiency-based training on simu-lators. Additional studies are warranted to furtheroptimize curriculum design and enhance acquisi-tion of durable skills.

REFERENCES1. Hamdorf JM, Hall JC. Acquiring surgical skills. Br J Surg

2000;87:28-37.2. Bridges M, Diamond DL. The financial impact of teaching

surgical residents in the operating room. Am J Surg 1999;177:28-32.

3. Scott DJ, Valentine RJ, Bergen PC, et al. Evaluating surgicalcompetency with the American Board of Surgery In-Training Examination, skill testing, and intraoperativeassessment. Surgery 2000;128:613-22.

4. Moore MJ, Bennett CL. The learning curve for laparoscopiccholecystectomy. The Southern Surgeons Club. Am J Surg1995;170:55-9.

5. Kohn LT, Corrigan JM, Donaldson MS. To err is human:building a safer health system. Washington (DC): NationalAcademy Press; 2000.

6. Jones DB, Brewer JD, Soper NJ. The influence of three-dimensional video systems on laparoscopic task perfor-mance. Surg Laparosc Endosc 1996;6:191-7.

7. Gallagher AG, McClure N, McGuigan J, Ritchie K, SheehyNP. An ergonomic analysis of the fulcrum effect in theacquisition of endoscopic skills. Endoscopy 1998;30:617-20.

8. Peters JH, Fried GM, Swanstrom LL, et al. Developmentand validation of a comprehensive program of education

SurgeryVolume 138, Number 2

Stefanidis et al 169

and assessment of the basic fundamentals of laparoscopicsurgery. Surgery 2004;135:21-7.

9. Fried GM, Feldman LS, Vassiliou MC, et al. Proving thevalue of simulation in laparoscopic surgery. Ann Surg 2004;240:518-25.

10. Scott DJ, Bergen PC, Rege RV, et al. Laparoscopic trainingon bench models: better and more cost effective thanoperating room experience? J Am Coll Surg 2000;191:272-83.

11. Hamilton EC, Scott DJ, Kapoor A, et al. Improving opera-tive performance using a laparoscopic hernia simulator. AmJ Surg 2001;182:725-8.

12. Seymour NE, Gallagher AG, Roman SA, et al. Virtual realitytraining improves operating room performance: resultsof a randomized, double-blinded study. Ann Surg 2002;236:458-63.

13. Brunner WC, Korndorffer JR, Sierra R, et al. Determiningstandards for laparoscopic competency using virtual reality.Am Surg 2005;71:29-35.

14. Korndorffer JR Jr, Scott DJ, Sierra R, et al. Developing andtesting competency levels for basic laparoscopic skills train-ing. Arch Surg 2005;140:80-4.

15. Arthur W Jr, Bennet WJ, Stanush PL. Factors that influenceskill decay and retention: a quantitative review and analysis.Human Performance 1998;11:57-101.

16. Schmidt RA, Bjork RA. New conceptualizations of practice:common principles in three paradigms suggest new con-cepts in training. Psychol Sci 1992;3:207-17.

17. Farr MJ. The long term retention of knowledge and skills:a cognitive and instructional perspective. New York:Springer Verlag; 1987.

18. Hurlock RE, Montague WE. Skill retention and its implica-tions for navy tasks: an analytical review. NPRDC SpecialRep. No. 82-21. San Diego (CA): Navy Personnel Reasearchand Development Center; 1982.

19. Schendel JD, Shields JL, Katz MS. Retention of motor skills:review (technical paper 313). Alexandria (VA): US ArmyResearch Institute for the Behavioral and Social Sciences;1978.

20. Schendel JD, Hagman JD. On sustaining procedural skillsover prolonged retention interval. J Appl Psychol 1982;67:605-10.

21. Annett J. Memory for skill. In: Gruneberg MM, Morris PE,editors. Applied problems in memory. London: Academic;1979.

22. Hikosaka O, Rand MK, Nakamura K, et al. Long-termretention of motor skill in macaque monkeys and humans.Exp Brain Res 2002;147:494-504.

23. Shadmehr R, Brashers-Krug T. Functional stages in theformation of human long-term motor memory. J Neurosci1997;17:409-19.

24. Bodilly S, Fernandez J, Kimbrough J, Purnell S. Individualready reserve skill retention and refresher training options.AD-A183416. Santa Monica (CA): Rand Corporation; 1986.

25. Grober ED, Hamstra SJ, Wanzel KR, et al. Laboratory basedtraining in urological microsurgery with bench modelsimulators: a randomized controlled trial evaluating thedurability of technical skill. J Urol 2004;172:378-81.

26. Torkington J, Smith SG, Rees B, Darzi A. The role of thebasic surgical skills course in the acquisition and retentionof laparoscopic skill. Surg Endosc 2001;15:1071-5.

27. Anastakis DJ, Wanzel KR, Brown MH, et al. Evaluating theeffectiveness of a 2-year curriculum in a surgical skillscenter. Am J Surg 2003;185(4):378-85.

28. Hamilton EC, Scott DJ, Fleming JB, et al. Comparison ofvideo trainer and virtual reality training systems on acqui-sition of laparoscopic skills. Surg Endosc 2002;16:406-11.

29. Scott DJ, Jones DB. Virtual reality training and teachingtools. In: Soper NJ, Swanstrom LL, Eubanks WS, editors.Mastery of endoscopic and laparoscopic surgery. Philadel-phia: Lippincott Williams & Wilkins; 2005. p. 146-60.

30. Korndorffer JR Jr, Clayton JL, Tesfay ST, et al. Multicenterconstruct validity for Southwestern laparoscopic video-trainer stations. J Surg Res. In press.

31. Kanfer R. Motivation theory and industrial and organiza-tional psychology. In: Dunnette MD, Hough LM, editors.Handbook of industrial and organizational psychology. PaloAlto (CA): Consulting Psychologists Press; 1992. p. 75-155.

SurgeryAugust 2005

170 Stefanidis et al