COMPASS All-hands Meeting, Fermilab, Sept. 17-18 2007 Scalable Solvers in Scalable Solvers in Petascale Electromagnetic Petascale Electromagnetic Simulation Simulation Lie-Quan (Rich) Lee, Volkan Akcelik, Ernesto Prudencio, Lixin Ge Stanford Linear Accelerator Center Xiaoye Li, Esmond Ng Lawrence Berkeley National Laboratory Work supported by DOE ASCR, BES & HEP Divisions under contract DE-AC02-76SF00515
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
COMPASS All-hands Meeting,
Fermilab, Sept. 17-18 2007
Scalable Solvers in Scalable Solvers in Petascale Electromagnetic Petascale Electromagnetic
SimulationSimulation
Lie-Quan (Rich) Lee, Volkan Akcelik, Ernesto Prudencio, Lixin Ge
Stanford Linear Accelerator Center
Xiaoye Li, Esmond NgLawrence Berkeley National Laboratory
Work supported by DOE ASCR, BES & HEP Divisions under contract DE-AC02-76SF00515
OverviewOverview
Shape Determination/Optimization V. Akcelik, L. Lee (SLAC) T. Tautges, P. Knupp, L. Diachin (ITAPS) O. Ghattas, E. Ng, D. Keyes (TOPS)
Linear and Nonlinear Eigensolvers L. Lee(SLAC), X. Li, E. Ng, C. Yang (LBNL/TOPS)
Scalable Linear Solvers L. Lee (SLAC), X. Li, E. Ng (TOPS)
Shape Determination Shape Determination
and Optimizationand Optimization
Shape Determination and Shape Determination and Optimization For SCRF Optimization For SCRF CavitiesCavities
Shape changes due to Fabrication errors Addition of stiffening rings Tuning for accelerating mode
Linear Solver is Linear Solver is Computational Kernel of Computational Kernel of Many CodesMany Codes Indefinite Matrices
Linear systems arising from shift-invert eigensolver in Omega3P
Indefinite linear system from KKT conditions S-parameter computation in S3P
Symmetric Positive Definite (SPD) Matrices From implicit time-stepping in T3P From thermal and mechanical analysis TEM3P From electro/magneto static analysis Gun3P
Issues in Petascale Electromagnetic simulations: Direct solver: memory usage, scalability of triangular solver Iterative solver: performance, effectiveness (preconditioner)
Omega3P Scalability on Omega3P Scalability on Jaguar/XTJaguar/XT with Iterative Linear with Iterative Linear SolverSolver
1.5M tetrahedral elements NDOFs = 9.6M NNZ = 506M
LCLS RF Gun
Scalability Using Sparse Direct Scalability Using Sparse Direct Solver MUMPSSolver MUMPS
Sparse Direct Solver is effective for highly indefinite matrices
Scalability affected by performance of Triangular Solver
N=2M, PSPASES Triangular Solver
N=2,019,968, nnz=32,024,600 No. of entries in L =1 billion
Need more scalable Triangular Solvers
More “More “Memory-usageMemory-usage” ” Scalable Sparse Direct Scalable Sparse Direct SolversSolvers
Maximal per-rank MU is 4-5 times than the average MU
Once it cannot fit into Nprocs, it most likely will not fit into 2*Nprocs
More “memory-usage” scalable solvers needed
MUMPS per-rank memory usage
N=1.11M, nnz=46.1M Complex matrix
Memory Saving Techniques Memory Saving Techniques
Single precision for factor matrix, iterative refinement to recover double precision accuracy (F)
Domain-specific Preconditioners Factorize real part of the matrix (R)
• Real part is a good approximation to the complex matrix User single precision to factorize real part of the
matrix (RF) Hierarchical preconditioners (FE order is the level)
(HP)• single precision for (1,1)-block (HPF)
• real part only for (1,1)-block (HPR)
• single precision & real part for (1,1)-block (HPRF)
Testing Results for Complex Testing Results for Complex Shifted Linear SystemsShifted Linear Systems
Recent Progress of Recent Progress of SuperLUSuperLU(Xiaoye Li)(Xiaoye Li)