Real-time surgery simulation of intracranial aneurysm ...edge.rit.edu/edge/P15083/public/Build Test Document... · Real-time surgery simulation of intracranial aneurysm ... Top row,

Real-time surgery simulation of intracranial aneurysmclipping with patient-specific geometries and haptic feedback

Wolfgang Fenz and Johannes Dirnberger

Research Unit Medical-Informatics, RISC Software GmbH, A-4232 Hagenberg, Austria

ABSTRACT

Providing suitable training for aspiring neurosurgeons is becoming more and more problematic. The increasingpopularity of the endovascular treatment of intracranial aneurysms leads to a lack of simple surgical situationsfor clipping operations, leaving mainly the complex cases, which present even experienced surgeons with achallenge. To alleviate this situation, we have developed a training simulator with haptic interaction allowingtrainees to practice virtual clipping surgeries on real patient-specific vessel geometries. By using specialized finiteelement (FEM) algorithms (fast finite element method, matrix condensation) combined with GPU acceleration,we can achieve the necessary frame rate for smooth real-time interaction with the detailed models needed fora realistic simulation of the vessel wall deformation caused by the clamping with surgical clips. Vessel wallgeometries for typical training scenarios were obtained from 3D-reconstructed medical image data, while for theinstruments (clipping forceps, various types of clips, suction tubes) we use models provided by manufacturerAesculap AG. Collisions between vessel and instruments have to be continuously detected and transformedinto corresponding boundary conditions and feedback forces, calculated using a contact plane method. Aftera training, the achieved result can be assessed based on various criteria, including a simulation of the residualblood flow into the aneurysm. Rigid models of the surgical access and surrounding brain tissue, plus coupling areal forceps to the haptic input device further increase the realism of the simulation.

Keywords: Intracranial aneurysm, Surgical clipping, Neurosurgery, Finite element method, Surgical simulation,Real-time simulation

1. INTRODUCTION

The demand for suitable training methods for aspiring neurosurgeons is getting bigger and bigger. The increasingpopularity of the endovascular treatment of intracranial aneurysms leads to a lack of simple surgical situations forclipping operations. Often only the complex aneurysms remain, which present even experienced surgeons with achallenge and are therefore not suitable for training purposes. While the usage of synthetic bio-models createdvia stereolithography or 3D printing can provide a remedy, it has many shortcomings, above all their complex andcostly production. Virtual surgery simulation can provide a much cheaper and versatile alternative. In additionto practicing with an unlimited number of specific aneurysm geometries multiple times, the neurosurgeon canassess the results of each training session (which are also stored in a database) according to different criteria,and thus determine which clip type, clip position or strategy is ideal for the particular aneurysm. Using existingCFD code, it is even possible to simulate the remaining blood flow after clamping the vessel/aneurysm with aclip.

While a number of real-time surgery simulators have existed for a time now in different fields of medicine (e.g. laparoscopy,1,2 cataract surgery,3 and tumor resection4,5), realistic virtual clipping surgery of intracranialvessels is still lacking. We present a system that accurately simulates clamping of thin vessel structures withrealistic deformations in real-time, using patient-specific data and accurate 3D models of surgical instrumentsand clips. Calculation of the residual blood flow through the clamped artery is also possible.

In the following sections, we will first describe the setup of the simulator and the generation of simulationmeshes from medical image data. Next, we will detail the real-time simulation of the deformable vessel structure,as well as the collision detection and response algorithms handling the contact between the models of the surgicalinstruments and the vessel. Finally, we will discuss the results and give a short conclusion.

Further author information:W. Fenz: E-mail: [email protected]

Figure 1. Left: View of a surgical clipping procedure. Right: Screenshot of the virtual clipping simulation.

2. METHODS

2.1 Simulation setup

The virtual surgery is performed with two Geomagic Touch (formerly Sensable Phantom Omni) haptic inputdevices. With the main hand, the operator controls the clipping forceps. Our simulator allows him to choosethe shape of the surgical clip attached to the forceps from a large set of different clip types. The opening angleof the clip is either controlled via the two buttons on the stylus of the Touch device, or via a real forceps thatcan be mounted to the device and is equipped with a Hall sensor that measures the opening angle of the forceps.The data from the sensor is obtained by USB cable via an analog-digital converter. With the other hand, thesurgeon can operate a secondary instrument such as a suction tube. During the simulation, it is possible to placemultiple clips on the vessel, pick up an already deposited clip again with the forceps, or exchange the equippedinstrument. With the secondary instrument, the trainee can for instance push vessels out of the way that blockaccess to the aneurysm, or, in case of the suction tube, also pull on arteries to move them. If the suction endis in contact with the vessel surface and the user presses the button on the haptic device, a permanent contactis generated that is separated again once the button is released. For perfomance reasons, the surrounding braintissue and vessels further away from the aneurysm are modeled as rigid meshes that provide haptic feedbackbut cannot be deformed. The system also supports stereoscopic 3D displays and LCD shutter glasses via theNVIDIA 3DVision API or Side-by-Side images. Support of the Oculus Rift Virtual Reality device is planned forthe near future.

2.2 Mesh generation

The finite element meshes we use for the deformable vessel walls are generated from real patient data. Utilizingan existing software package we developed in an earlier project (MEDVIS 3D6), a 3D reconstruction of a CTimage set is performed, and a region of interest is chosen in the resulting voxel volume. Since these imagesusually only show the (contrast enhanced) blood lumen, we have to generate the wall mesh via the blood surfacemesh (Fig. 2). After calculation of the vessel centerline, definition of inlet and outlet planes, and choosing anintensity threshold, an isosurface of the blood lumen is created using an adaptive skeleton climbing algorithm.7

This preliminary surface mesh is then cut at the inlets and outlets, smoothed and remeshed with NETGEN.8

We then create a three-dimensional orthogonal grid around it, and calculate for each point close to the surfaceits minimal distance to the mesh. Using this distance field as our new voxel volume, we can repeat the previousprocedure (with a specified wall thickness as threshold value) and obtain the outer surface mesh of the vesselwall. By weighting the distance with a factor depending on the location of the closest mesh vertex, we can alsolet the thickness vary between two predefined values (one for the arterial wall and one for the aneurysm wall,

Figure 2. Generation of the vessel wall mesh from medical image data. Top row, left: One slice of a series of imagesobtained from CT angiography. Top row, right: 3D-reconstruction of the intracranial vessel tree. Middle row, left:Region of interest obtained from the reconstructed voxel volume, including a saccular aneurysm and its parent vessel.The calculated center line (voxel-based) and the inflow (blue) and outflow (green) planes are also shown. Middle row,right: Triangular surface mesh of the corresponding blood volume (smoothed and remeshed with 0.4 mm resolution).The aneurysm surface (dark green) is detected automatically. Bottom row, left: Wall mesh generated with the distancefield method described in the text (ca. 15,000 elements, 5,000 nodes). Bottom row, right: Cross section through the wallmesh, showing the varying wall thickness (0.4 mm around the parent vessel, two thirds less around the aneurysm).

Figure 3. 3D models of a typical surgical clip (left) and forceps (right) used in the simulation (data courtesy of AesculapAG).

which is usually thinner), see Fig. 2. All that remains now is to fill the cross-sectional area of the wall withtriangles and create a tetrahedral volume mesh, for which we use the TETGEN mesh generator.9

Surface meshes of instruments and clips have either been provided as CAD models by our project partnerAesculap AG, a leading manufacturer of surgical clips and instruments, or manually modeled according to theirproduct catalog (see Fig. 3). Opening/closing animations for the clip models were generated with AutodeskMaya 3D.

2.3 Deformable mesh simulation

Due to the complex geometry of cerebral aneurysms and their relatively thin vessel wall structure, in our experi-ence the deformable simulation mesh has to consist of at least 10,000 elements. In a first approach, we have triedusing finite element models for nonlinear elasticity (corotational FEM,10 TLED11), which are more accurate incase of large deformations, but more demanding in terms of computation times. Our tests have shown thattheir performance was not sufficient for a real-time simulation of meshes with this resolution. We have thereforedecided to use a linear elasticity model for the time being, while keeping in mind that with increasing hardwarecapabilities, a nonlinear model could very well become feasible in the near future.

In linear elasticity, the displacement d(r, t) of an elastic body is obtained by solving the Navier-Lame equation,

(λ+ µ)∇(∇d) + µ∇2d + F = ρ∂2d

∂t2, (1)

where F is the vector of external forces, λ, µ are the Lame material parameters, and ρ is the density. Applyingthe FEM on a given computational mesh to (1) yields the matrix equation

Kd = f , (2)

where K is the stiffness matrix of the system, d the vector of nodal displacements, and f the nodal vectorof applied force. A number of methods exist to speed up such calculations for real-time simulations. Mostimportantly, it is possible to calculate the inverse of the global system matrix K, which depends solely on thematerial parameters and the geometry, prior to the simulation, and then solve for the nodal displacements bymultiplying the current load vector with the inverse system matrix,

d = K−1f =∑i

K−1i fi. (3)

Here, K−1i is the i-th row of K−1 and fi the i-th component of f . For a sparse load vector, this computation

can be done very fast (Selective Matrix Vector Multiplication, SMVM12), meaning that only terms with fi 6= 0are taken into account in calculating (3). Furthermore, the dimension of the system can be reduced by so-called

matrix condensation, which transforms the original equation system into a smaller one containing only the visibledegrees of freedom (those belonging to the vertices on the outer surface of the mesh). In contrast to the originalstiffness matrix, the condensed matrix is not sparse any more, however, this is not an issue since we calculatethe inverse only once in the pre-simulation stage. The combination of the described techniques is known as fastfinite element (FFE) Method.12–14

Special care must also be taken regarding the boundary conditions: Since we obtain displacements insteadof forces from the collisions between an instrument and the vessel wall, we first have to transform these intocorresponding forces that enter into the load vector f in (2). This is done by solving a linear system of dimension3n, where n is the number of displaced nodes.1 For fine meshes, when a clip is closed around an aneurysm, thenumber of colliding nodes n can become rather high (one hundred or more), therefore it is essential to use afast solver for this calculation. If no time dependence is considered in the elasticity equation (1), these collisionforces are the only appearing entries in the load vector, so it remains sparse. Since oscillations are not significantin our application, and the visual difference is minimal, we follow this route and solve the static equation only.

Once a clip has been clamped onto the vessel, it loses its connection with the forceps and remains fixed in itsplace, and with it the nodes of the vessel it is displacing. The trainee can now equip the forceps with anotherclip and interact with the vessel again. When deforming the artery, however, the first clip should now movetogether with the vessel. In order to achieve this effect, we solve the elasticity equation twice in this case. First,we solve it without the boundary conditions due to the first clip, and look at the set of nodes it was displacing,calculating the translation of their center of gravity from its resting position. This vector is then added tothe fixed displacements caused by the first clip when solving the system equation once again, this time takinginto account both clips. Though not strictly accurate, this method yields a visually convincing result. Thecomputational time increases of course now, and thus the frame rate of the simulation is somewhat diminished.

It should be noted that the elastic properties of intracranial vessel and aneurysm walls that appear in (1) aredifficult to measure and therefore not accurately known. The values of Young’s modulus E found in literaturerange between 1 and 300 MPa.15–17 Since we are currently using a constant modulus for the whole mesh includingartery and aneurysm wall in our simulations, we chose a value of E = 1 MPa and ν = 0.45 for the Poisson ratio.We plan to incorporate the possibility of assigning different values of E to different parts of the simulation meshin the future.

The calculation of the FEM stiffness matrix and the SMVM calculations are implemented using the CUDAGPU architecture in order to achieve real-time performance. The matrix equation for obtaining the boundaryforces is also solved on the GPU via a parallel algebraic multigrid algorithm.18

2.4 Collision detection and response

The collision detection (CD) problem for our simulation setup is complicated considerably by the fact that thevessel wall is deformable. We use the Bullet Physics Library,19 a fast and well tested open source collisiondetection library, as a basic tool to find contacts between objects in our scene, represented by triangular collisionmeshes. For the vessel wall, only the outer surface is taken into account. In case of an overlap between aninstrument/clip and the vessel, Bullet yields a set of collision points and interpenetration vectors. For eachcollision point, the closest-lying vessel surface node and, to a lesser extent, its nearest neighbor nodes, areassigned a fixed displacement according to the penetration depth. This amounts to a set of boundary conditionsfor the vessel deformation. Solving the FEM equation system for the wall elasticity yields the correspondingdisplacements of the remaining nodes, and leads to a new wall shape. The vertices of the associated collisionsurface must then be updated accordingly. Since the collision points in general do not coincide with a meshnode, one has to specify how the collision vector is transformed into a nodal displacement. We achieved the bestresults using the normal vector of the nearest mesh node as the direction and about half the penetration depthas the length of the displacement vector.

The force feedback vector to be sent to the haptic device also has to be chosen carefully. Using the sameforce vector as in the FEM equation does not work well, since the spatial and temporal discretization leads todiscontinuities that result in oscillations or abrupt movements of the haptic device. It is therefore essential tointerpolate smoothly between consecutive force values. We found that, in our opinion, a better solution is the

contact plane collision response method.20 Here, one takes the tangential plane int the point of the first collisionas reference, and uses its normal vector as the direction of the feedback force for as long as the meshes arein contact. The magnitude of the force vector is set proportional to how deep the instrument has penetratedthe plane. This way, the force vector always changes continuously. The drawback of this approach is that thedirection of the force does not change as long as the collision persists, even if the actual physical force wouldalready differ significantly from the original direction (e. g. if the instrument slides along a curved surface).Friction is also simulated by adding a force proportional to the negative instrument velocity component parallelto the contact plane.

If several spatially divided points of contact between vessel and instrument exist, multiple contact planes arecreated, and the force vectors corresponding to each of them are summed up. Internally, this is implementedwith so-called contact areas, consisting of the vertices of the deformable surface currently in collision plus theirsurrounding neighbor vertices. Thus, if a new nodal contact is established, two cases can be distinguished.If the instrument has moved along the vessel surface or the surface area in contact with the clip has grownbecause the clip is closed, the new collision nodes are already part of an existing contact area, which is enlargedcorrespondingly. In the other case, e. g. if the second arm of the clip or the secondary instrument make additionalcontact, they become the seeds of a new contact area. Whenever an area loses contact (the instrument is movedaway from the vessel), its vertices are moved continuously to their initial position, until they either reach it orcollide with the instrument again. This way, the vessel moves realistically with the instrument, if it is slowlymoved back after deforming the vessel.

To speed up the simulation, the CD, FEM and visualization tasks run in parallel threads, each having aseparate copy of the deformable mesh, that are synchronized whenever changes occur. For example, the collisionmesh of the vessel changes if a colliding clip is moved towards the vessel, increasing the displacement of thecontact nodes. Once the displacements have been updated, the change is transferred to the FEM thread in theform of new boundary conditions. The prescribed displacements are then converted to forces as per section 2.3,and the new displacement field is obtained via Eq. (3). While the FEM thread is calculating, the CD threadcontinues to track the contact and update the collision mesh, so that the FEM calculation does not impede thecontinuous detection of fast-changing displacements. Once the FEM calculation is finished, the nodes of thecollision mesh that are not in contact with the instrument are assigned their new positions, while the updatedcontact displacements are again transferred to the FEM thread. Similarly, the calculated displacement field issent to the visualization thread, which updates the copy of the surface mesh that is used for rendering.

3. RESULTS

Using the described finite element method, real-time simulation of surgical clipping with large meshes on medium-range hardware becomes feasible. We have tested meshes with up to 35,000 elements (11,000 nodes) on an desktopPC with an Intel i7-3930K CPU, 8GB RAM and an Nvidia GeForce GTX 580, and the frame rate was still above20 fps. The pre-simulation stage including condensation and inversion of the system matrix takes less than30 minutes on a single processor (using the LU-decomposition method of the Eigen library21). The originaluncondensed global system matrix of the largest mesh we considered takes up 4.6 GB of memory (using singleprecision floating point numbers), the condensed matrix (16,500 degrees of freedom) still about 1 GB. Takingadvantage of the symmetry of the stiffness matrix, we can, however, reduce the storage size by approximatelyone half. For each aneurysm model, once the inverse of the condensed system matrix is calculated, it is storedtogether with the degrees of freedom corresponding to the matrix rows, and the next time the same model issimulated, the matrix is loaded automatically.

Once a training session is ended, the results of the surgery can be assessed according to different criteria.These include the total time needed, the number of clips used, how often clips were repositioned, and the numberof times a certain force or velocity threshold was exceeded. Another important result is the remaining blood flowinto the aneurysm, if there is any. Using code we developed in a previous project (MEDVIS 3D6), we can performa blood flow simulation through the interior of the deformed vessel wall. Since we do not check for self-collisionsof the wall mesh during the real-time simulation, clamping the vessel usually leads to intersections of the innersurface of the wall (see Fig. 4, top left). Before a simulation can be started, therefore, these self-intersectingregions have to be removed, leaving only the volume that can be reached by the blood flow. Similar to Ref. 22,

Figure 4. Removal of self-intersections of the inner wall surface after clipping: Original intersecting mesh (top left), afterremoval of invalid triangles (top right), after smoothing of the resulting hole (bottom left), final closed and remeshedblood surface (bottom right).

Figure 5. Results of a blood flow simulation after clipping of the aneurysm. The figures show streamlines of the velocityfield (top) and the wall shear stress on the surface of the blood volume (bottom).

we apply a region growing algorithm in order to determine the valid region of the surface mesh, using the methodof Moller23 to detect intersecting triangles. Once the valid region is found, all other triangles are removed fromthe mesh, leaving us with a surface that has no self-intersections, but a hole with a jagged boundary where thecollisions occurred (Fig. 4, top right). After smoothing the boundary, the hole is filled using a method includedin the MeshLab24 library. After remeshing the resulting closed surface with a higher resolution and generatinga tetrahedral volume mesh, the mesh can finally be used for a CFD simulation. Currently, we calculate thestationary flow through the clamped artery for a given inflow velocity, using standard values for blood viscosityand density. The user can then visualize the velocity field with vectors or streamlines, as well as the pressureand wall shear stress distribution on the surface (see Fig. 5).

4. CONCLUSION

In close cooperation with the neurosurgery department of Wagner Jauregg hospital (Linz, Austria) and surgicalclip manufacturer Aesculap AG, we have developed a physically accurate clipping surgery simulator with haptic

feedback and real-time performance. It supports placing and repositioning of a variety of different surgical cliptypes and the use of a secondary instrument (e. g. suction tube) in the off hand. Surgeries can be trained ondifferent vessel geometries obtained from patient-specific image data, and training assessment is possible usingvarious criteria ranging from simple ones like the number of clamping attempts to a detailed calculation of theremaining blood flow in the post-surgical geometry. All results of a training are stored in a database, and canbe viewed again at a later point.

For the future, we plan to improve our simulator by inclusion of the surgical planning phase (choice of entry,preparation), and more realistic vessel properties (locally varying elasticity parameters, nonlinear models).

ACKNOWLEDGMENTS

This work was funded in the framework of the BRIDGE program of the Austrian Research Promotion Agency(FFG), project number 838519.

REFERENCES

[1] Kuhnapfel, U., Cakmak, H., and Maaß, H., “Endoscopic surgery training using virtual reality and deformabletissue simulation,” Computers & Graphics 24, 671–682 (Oct. 2000).

[2] Lamata, P., Gomez, E. J., Sanchez-Margallo, F. M., and Lopez, O., “SINERGIA laparoscopic virtual realitysimulator: Didactic design and technical development,” Computer Methods and Programs in Biomedicine 85,273–283 (2007).

[3] Choi, K.-S., Soo, S., and Chung, F.-L., “A virtual training simulator for learning cataract surgery withphacoemulsification,” Computers in Biology and Medicine 39, 1020–1031 (Nov. 2009).

[4] Bambakidis, N. C., Selman, W. R., and Sloan, A. E., “Surgical rehearsal platform: potential uses inmicrosurgery,” Neurosurgery 73 Suppl 1, 122–126 (Oct. 2013).

[5] Sato D., Kobayashi R., Kobayashi A., Fujino S., and Uchiyama M., “Soft tissue pushing operation using ahaptic interface for simulation of brain tumor resection,” Journal of Robotics and Mechatronics 18(5), 634(2006).

[6] Fenz, W. and Dirnberger, J., “Patient-specific blood flow simulation to improve intracranial aneurysmdiagnosis,” in [Proc. SPIE 7964 ], Wong, K. H. and Holmes III, D. R., eds., 79642D–79642D–9 (Mar. 2011).

[7] Poston, T., Wong, T.-T., and Heng, P.-A., “Multiresolution isosurface extraction with adaptive skeletonclimbing,” Computer Graphics Forum 17(3), 137–147 (1998).

[8] Schoberl, J., “NETGEN an advancing front 2d/3d-mesh generator based on abstract rules,” Computing andVisualization in Science 1(1), 41–52 (1997).

[9] Si, H., “TetGen: A quality tetrahedral mesh generator and three-dimensional delaunay triangulator,” (2013).http://tetgen.berlios.de/.

[10] Muller, M., Dorsey, J., McMillan, L., Jagnow, R., and Cutler, B., “Stable real-time deformations,” in [Proc.SIGGRAPH Symposium on Computer Animation (SCA) ], 49–54 (2002).

[11] Miller, K., Joldes, G., Lance, D., and Wittek, A., “Total lagrangian explicit dynamics finite element algo-rithm for computing soft tissue deformation,” Communications in Numerical Methods in Engineering 23,121–134 (Aug. 2006).

[12] Bro-Nielsen, M., “Surgery simulation using fast finite elements,” in [Visualization in Biomedical Computing ],Goos, G., Hartmanis, J., van Leeuwen, J., Hohne, K. H., and Kikinis, R., eds., 1131, 529–534, SpringerBerlin Heidelberg, Berlin, Heidelberg (1996).

[13] Bro-Nielsen, M., “Finite element modeling in surgery simulation,” Proceedings of the IEEE 86, 490–503(Mar. 1998).

[14] Berkley, J., Turkiyyah, G., Berg, D., Ganter, M., and Weghorst, S., “Real-time finite element modeling forsurgery simulation: An application to virtual suturing,” IEEE Transactions on Visualization and ComputerGraphics 10, 314–325 (May 2004).

[15] Torii, R., Oshima, M., Kobayashi, T., Takagi, K., and Tezduyar, T. E., “Influence of wall elasticity inpatient-specific hemodynamic simulations,” Computers & Fluids 36, 160–168 (Jan. 2007).

[16] Torii, R., Oshima, M., Kobayashi, T., Takagi, K., and Tezduyar, T. E., “Numerical investigation of the effectof hypertensive blood pressure on cerebral aneurysm—dependence of the effect on the aneurysm shape,”International Journal for Numerical Methods in Fluids 54, 995–1009 (June 2007).

[17] Ahmed, S., Sutalo, I. D., Kavnoudias, H., and Madan, A., “Fluid structure interaction modelling of a patientspecific cerebral aneurysm: Effect of hypertension and modulus of elasticity,” in [16th Australasian FluidMechanics Conference (AFMC) ], 75–81, School of Engineering, The University of Queensland, Brisbane,Australia (Dec. 2007).

[18] Haase, G., Liebmann, M., Douglas, C., and Plank, G., “A parallel algebraic multigrid solver on graphicsprocessing units,” in [High Performance Computing and Applications ], Zhang, W., Chen, Z., Douglas, C.,and Tong, W., eds., Lecture Notes in Computer Science 5938, 38–47, Springer Berlin Heidelberg (2010).

[19] “Real-time physics simulation,” (2015). http://bulletphysics.org.

[20] Chan, L. S.-H. and Kup-Sze Choi, “Integrating PhysX and OpenHaptics: Efficient force feedback generationusing physics engine and haptic devices,” 853–858, IEEE, Tamsui, Taipei (Dec. 2009).

[21] Guennebaud, G., Jacob, B., and others, “Eigen v3,” (2010). http://eigen.tuxfamily.org.

[22] Jung, W., Shin, H., and Choi, B. K., “Self-intersection removal in triangular mesh offsetting,” Computer-Aided Design and Applications 1, 477–484 (Jan. 2004).

[23] Moller, T., “A fast triangle-triangle intersection test,” Journal of Graphics Tools 2, 25–30 (1997).

[24] Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., and Ranzuglia, G., “Meshlab: an open-source mesh processing tool,” in [Eurographics Italian Chapter Conference ], 129–136, The EurographicsAssociation (2008).