SciDAC TOPS PETSc Work

1

SciDAC TOPS PETSc Work

http://www.mcs.anl.gov/[email protected]

SciDAC TOPS DevelopersSatish Balay

Chris BuschelmanMatt KnepleyBarry SmithHong Zhang

Other PETSc Team MembersLois Curfman McInnes, Bill Gropp, Dinesh Kaushik

2

PETSc

• Portable, Extensible, Software for the scalable solution of nonlinear PDEs

• Focusing on algebraic solvers– Preconditioners, multigrid, Krylov methods,

Newton, FAS, time integrators

3

Activities• Derivative calculations – ADIC/ADIFOR

– Matrix-free preconditioners– FAS (future)

• Accessing other software– HYPRE preconditioner usage– RAMG,SAMG– SuperLU– ESI

• SIDL/Babel/CCA – PETSc 3.0 plans• SIDL/Babel MPI

4

Derivatives

• Automatic tools ADIC/ADIFOR are great but require manual processing (painful when developing code)

• Would like to have a variety of derivatives “for free” (or nearly)– Matrix element based Newton which requires explicit

sparse Jacobian– Matrix-free Newton– SOR preconditioners and FAS require one equation at

at time

5

To Fully Automate

• Code comments to indicate what needs to be processed

• Predefined calling sequences

• Libraries that can use generated code

• Make system automatically generates required derivative code

6

../* Process adiC: FFL */int FFL (DALocalInfo *info,Field **x, Field **f,void *ptr) { AppCtx *user = (AppCtx*)ptr; ..

DMMGSetSNESLocal(dmmg,FFL,0,ad_FFL,admf_FFL);

typedef struct { PassiveReal lidvelocity,prandtl,grashof; PetscTruth draw_contours;} AppCtx;

typedef struct { double u,v,omega,temp;} Field;

7

> make BOPT=g_c++ ex19Running adiC on function FFLExtracting structure FieldExtracting structure AppCtxExtracting function FFL

adiC -a -d gradient -i /sandbox/bsmith/petsc/bmake/adic.init ex19.c.tmpADIC 1.1 beta 4 08/07/00 18:06:31This software was created in the course of academic and/or researchendeavors by Argonne National Laboratory. See the disclaimer in the

g++ -fPIC -o ex19.o -c -Dad_GRAD_MAX=36 -I/sandbox/bsmith/petsc/include/adic -g -I/sandbox/bsmith/petsc ..

8

Matrix-free Multiply

• Differencing (well understood)

• AD

int mf_FFL(DALocalInfo *info,mf_Field **x, mf_Field **f,void *ptr) {

typedef struct { double value;double grad[1];} mfDERIV_TYPE;typedef struct { mfDERIV_TYPE u, v, omega, temp;}

9

Matrix-free Multigrid Preconditioners

• Krylov smoothing with – no preconditioner– (point block) Jacobi preconditioner– (point block) SOR preconditioning

10

Matrix-free Jacobi/SOR

• If function is provided an “(block) equation at a time” matrix-free multiply can be done an equation at a time– can be computed by providing a seed of – For SOR

• Similar computations with differencing

iiJ ie1

:( )i i ii i iu u J b J u

11

FAS – Full Approximation Scheme (Nonlinear Multigrid)

• Smoothing uses Newton to solve a single (block) equation at a time– Need the same “(block) equation at a time” and

its derivative as for matrix-free Newton!

12

We are developing a common software infrastructure and user interface for many Newton methods and FASwhich can use AD or differencing for derivatives.

13



• Accessing other software– HYPRE preconditioner usage– RAMG, SAMG– SuperLU– ESI


14

Philosophy on Solvers

• Use any solver without changing the application code

• Add to application code to take advantage of special solver features (optimizations)– but additions cannot break other solvers

PCILUSetDropTolerance(pc,1.e-5);PCSPAISetEpsilon(pc,1.e-2);

15

HYPRE-PETSc Matrix Element based preconditioners

• No vector copies

• Currently a matrix copy (plan to remove)

-pc_type hypre -pc_hypre_type <pilut,parasails,boomerAMG,euclid>-pc_hypre_pilut_tol 1.e-4-pc_hypre_parasails_thresh .1-pc_hypre_parasails_filter .2

16

RAMG, SAMG – PETScSequential Preconditioners

• AMG – written by John Ruge, Klaus Stueben, and Rolf Hempel. A classic scalar PDE AMG code.

• SAMG – written by Klaus Stueben. A complete rewrite that supports systems of PDEs (interface coming soon).– PETSc interfaces provided by Domenico

Lahaye

17

SuperLU_Dist• MPI parallel LU factorization and solve written by Sherry Li and Jim Demmel

• Henry Tufo – Paul Fischer XYT fast direct parallel solver for “small problems”, i.e. the coarse grid problem in multigrid

-mat_aij_superlu_dist-mat_aij_superlu_dist_colperm <MMD_AT_PLUS_A,NATURAL,MMD_ATA,COLAMD>

18

ESI Equation Solver Interface

• A C++ “standard” of abstract classes for setting up and solving linear systems

• At least three implementations– Trilinos (Sandia, NM)– ISIS (Sandia, CA)– PETSc

19

PETSc ESI Goal

• Allow PETSc to use matrices and solvers written by others in ESI

• Allow anyone to use PETSc matrices and solvers from ESI– Achieved by using two sets of wrappers

• PETSc objects “look like” ESI objects

• ESI objects “look like” PETSc objects

20

-is_type esi –is_esi_type Petra_ESI_IndexSpace-vec_type esi -vec_esi_type Petra_ESI_Vector -mat_type esi –mat_esi_type Petra_ESI_CSR_Matrix

PETSc using Trilinos objects

ESI using PETSc objects

esi::Vector<double,int> *v = new esi::petsc::Vector<double,int>(petscvector)esi::Operator<double,int> *o = new esi::petsc::Matrix<double,int>(petscmatrix)

21





22

SIDL/Babel (LLNL)

• Scientific Interface Definition Language

• Babel – generates language stubs for– C, C++, Fortran 77, Python, Java, …

• Provides – Object model– Exception (error) model

• Immature, but very exciting development

23

PETSc 3.0

• SIDL/Babel for inter-language, object model and exception model

• Multiple precisions – Single for preconditioners?

• Vector and matrix sizes mutable– Adaptive mesh refinement

• Not a rewrite, just a rewrap!

24





25

Conclusions

• Many new tools and solvers

• Accessible via – PETSc 2.0 interface– ESI– (coming) SIDL/Babel/CCA

• We are happy to support users, especially those interested in nontrivial problems

SciDAC TOPS PETSc Work

Documents