Fred a new GPU-based fast-MC code and its applications in proton beam therapy MCMA 2017 - Napoli A. Schiavi
Fred a new GPU-based fast-MC code
and its applications in proton beam therapy
MCMA 2017 - Napoli
A. Schiavi
Fast paRticle thErapy Dose evaluator
Collaboration• A. Schiavi, V. Patera, M. Senzacqua, Univ. La Sapienza
Roma /INFN (Italy)
• - G. Battistoni, S. Pioli - INFN (Italy)
• - I. Rinaldi, N. Krah - CNRS/IN2P3 and Lyon 1 University (France)
• A. Rucinski, J. Gajewski - PAN, Krakow (Poland)
FRED fast-MC platform• MC for protons in voxel geometry
• Tabulated total stopping power in water (PSTAR-NIST), energy straggling (Gaussian and Landau-Vavilov regimes)
• MCS models: single-,double-,triple-gaussian, 2 gauss+Rutherford
• Nuclear interactions: elastic and inelastic; fragmentation; local deposition of heavy ions; tracking of secondary protons and deuterons
• HU to density conversion (Schneider) and stoppow calibration
• MC-TPS: dose optimization using DDO (Lomax)
• RBE models = fixed 1.1, LETd-based (Wedenberg, Carabe, Wilkens, Chen), table-based (LEM1, MKMPIDE)
Fred - no nucl
Fred
Fluka
200 MeV protons in liquid waterDose map of a pencil beam
Water model: energy deposition
100 MeV 150 MeV 200 MeV 250 MeV
nucl. off
elasticnucl. only
elasticnucl. only
nucl. on
GeantFLUKAFred
22
20
18
16
14
1224 25 26 27
A. Schiavi et al, PMB 62 (2017) 7482–7504
Longitudinal profile and lateral tails
c)
d)
a)
Dose map
FLUKA
b)
QA SOBPz x
y
head-onside-on
Water tank CT
3 cm cube at 15 cm depth
QA SOBP: dose profiles and measurements
E = 226.61 MeV/u at 20 cm depth
Field size factor
• standard codes
• expensive (€€€€€)
• maintenance (staff)
• low budget (€€€€)
• redundancy
• in-house maintenance
+
Hardware for rapid MC recalculation
Parallel execution model in Fred
Fred front-end
ExtranodeMPI
IntranodeMulti-GPUOpenCl
IntranodePOSIX
Multi-threads
Queues and timeline
Execution timeline for 8 queues on 4 GPUs using OpenCL.
Host-to-device transfers (green), kernel execution (red), and device-to-host (blue) transfers
Hardware and Performance
full-MC * * FLUKA or Geant4C
PU
benchmark: 150 MeV protons in a water phantom on a 1 mm3 dose scoring grid
Hardware and Performance
full-MC * * FLUKA or Geant4C
PU
GP
U
• Patient-specific QA protocol at CNAO
• Patient-specific HU-RSP calibration
• Commissioning of CCB proton center in Krakow
• Dose monitoring using secondary protons
Applications to proton therapy
see S. Muraro talk this afternoon (ID 67)
Fast-MC recalculation of patient verification plans at CNAO
Fast-MC recalculation of patient verification plans at CNAO
18
Patient verification planTPS
FRED
TPS
FRED
Gy
tps
fred
head-on side-on
fred
tps
99.6% @ 2mm/2% 96.7% @ 1mm/1%
ɣ-index pass rates
Patient-specific and treatment-specific HU-RSP
calibration
Fred
Fred commissioning @ CCB Krakow Proton Beam Therapy Centre
FRED central cross-section
0 50 100 150 200 250
Z [mm]
-30
-20
-10
0
10
20
30
X [m
m]
0 50 100 150 200 250
Z [mm]
0
0.2
0.4
0.6
0.8
1
Dose
norm
. [-
]
Bragg Peak comparison
TPS Beam Model
(Rmax
=156.59mm, R 2
bort=1.000)
FRED MC sim.
(Rmax
=156.59mm, R 2
bort=1.000)
0 50 100 150 200 250
Z [mm]
2
3
4
5
6
7
Spot si
ze [m
m]
Spot primary sigma at depth
TPS σmean
Prim
FRED σmean
Prim
0 50 100 150 200 250
Z [mm]
5
10
15
20
25
Spot si
ze [m
m]
Spot secondary sigma at depth
TPS σmean
Sec
FRED σmean
Sec
-50 -40 -30 -20 -10 0 10 20 30 40 50
X [mm]
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Dose
norm
. [-
]
σXTPS=5.61mm (R 2=0.9999)
σXFRED=5.65mm (R 2=0.9999)
TPS Eclipse profile
FRED MC sim. profile
Signle Gauss fit to TPS
Signle Gauss fit to FRED
-50 -40 -30 -20 -10 0 10 20 30 40 50
X [mm]
10 -6
10 -5
10 -4
10 -3
10 -2
10 -1
10 0
Log d
ose
norm
. [-
]
Prim. σXTPS=5.74mm
Sec. σXTPS=16.50mm
(R2=0.9994)
Prim. σXFRED=5.55mm
Sec. σXFRED=11.06mm
(R2=0.9943)
TPS Eclipse profile
FRED MC sim. profile
Double Gauss fit to TPS
Double Gauss fit to FRED
-50 -40 -30 -20 -10 0 10 20 30 40 50
Y [mm]
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Dose
norm
. [-
]
σYTPS=5.19mm (R 2=0.9999)
σYFRED=5.54mm (R 2=0.9999)
TPS Eclipse profile
FRED MC sim. profile
Signle Gauss fit to TPS
Signle Gauss fit to FRED
-50 -40 -30 -20 -10 0 10 20 30 40 50
Y [mm]
10 -6
10 -5
10 -4
10 -3
10 -2
10 -1
10 0
Log d
ose
norm
. [-
]
Prim. σYTPS=5.38mm
Sec. σYTPS=16.54mm
(R2=0.9993)
Prim. σYFRED=5.47mm
Sec. σYFRED=11.08mm
(R2=0.9950)
TPS Eclipse profile
FRED MC sim. profile
Double Gauss fit to TPS
Double Gauss fit to FRED
Fred code is currently being commissioned at CCB as a quality assurance tool.Preliminary results show good agreement of single beam dose distributions calculated with Eclipse and Fred, indicating an accurate implementation of CCB beam model in the Fred MC-TPS code. Dose distributions for a complete plan can be obtained in about one minute using Fred on GPU. Robustness studies of treatment plan strategy can be conducted on the HPC cluster Prometheus.
Top performance on Prometheus
• 72 Nodes with 24 CPUs and 2 Tesla K40d GPUs
• Up to 144 GPUs in parallel + 1728 CPUs
perfect linear scaling
up to 0.3 billion primary/s
2.4 mln/s per GPU
This research was supported in part by PL-Grid Infrastructure.
• clinical validation of fast-recalculation tool
• applications to clinical routine
• extensions to include other ions (Carbon, Helium) and secondary particles (alphas, delta-rays and neutrons)
• dose monitoring using charged secondary particles
Future developmentsand perspectives
24
Patient recalculation plan
tps
fred
gamma-index 97% @ 2mm/2% gamma-index 92% @ 1mm/1%
recalculation at 1% = 700 million primary protons
simulation time = 72 s
Hardware: 4x GPU NVIDIA GTX 10801x CPU Intel i7-5930K @ 3,50 GHz with 12 cores
20 mln primary/s
Water-cooled 4 GPU workstation
4x NVIDIA Titan-Xp40 mln primary/s
budget: 10 kEuro
compare with newNVIDIA DGX-1(8x Tesla P100)
expected performance:80 mln primary/s
budget: 125 kEuro
Case study: 3D raytracing for legacy F77 hydrocode
From 1 to 2 Mray/s(equivalent to 800 MPI processes)
Raytracing step well belowhydrodynamic step
Duty-cycle 30%
Duty-cycle 100%
4 water-cooled GPU
2 air-cooled GPU