Environmental Molecular Sciences Laboratory Pacific Northwest National Laboratory SPSCICOMP2000 8/15/2000 EA Edoardo Aprà High Performance Computational Chemistry Environmental Molecular Sciences Laboratory Pacific Northwest National Laboratory Richland, WA 99352 NWChem software development with Global Arrays
NWChem software development with Global Arrays. Edoardo Apr à High Performance Computational Chemistry Environmental Molecular Sciences Laboratory Pacific Northwest National Laboratory Richland, WA 99352. Outline. Global Arrays Toolkit overview NWChem overview Performance - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
Intelligent Protocols in ARMCI
On IBM SP with SMP nodes ARMCI exploits • cluster locality information• Active Messages• remote memory copy• shared memory within SMP node• threads
0
20
40
60
80
100
120
140
1 100 10000 1000000
bytes
band
wid
th [M
B/s]
ARMCI
Lapi remote
Lapi SMP
P0 P1 P2 P3adapter
memcpy
P0 P1 P2 P3
adapterSMP node
put
LAPILAPI
logically partitioned shared memory segment
local process memory
swit
chSMP node
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
Advanced networks
Exciting research and development opportunities High-performance <10S latency, 100-200 MB/s bandwidth
Low cost
Flexible
Finally the traditionally closed interfaces open
Protocols: GM(Myrinet), VIA (Giganet), Elan (Quadrics) offer a lot of capabilities and performance to support not only MPI but also more advanced models
H/W support for 1-sided communication
NIC support More of the high-level protocols pushed down to h/w
Opportunities for optimization e.g., fast collective operations
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
High Performance Parallel I/O Models
Disk Resident Arraysarray objects on disk
(RI-SCF, RI-MP2)
Exclusive Access Filesprivate files per processor
(semidirect SCF and MP-2)
Shared Files“shared-memory on disk”
(MRCI)
ELIOdevice library
Distant I/O one-sided
communication to disk
portability layer
applicationlayer
hardware
filesystem layer
System of Hints allows performance tuning to match application characteristics
filesystem Cfilesystem B
filesystem A
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
Forms of Parallel I/O in Chemistry Apps
collectiveto shared file
independentto private files
independentto shared file
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
Full eigensolution performed on a matrix 966x966
Unique feature not available elsewhere• Inverse iteration using Dhillon-Fann-
Parlett’s parallel algorithm (fastest uniprocessor performance and good parallel scaling)
• Guaranteed orthonormal eigenvectors in the presence of large clusters of degenerate eigenvalues
• Packed Storage• Smaller scratch space requirements
0
10
20
30
40
50
60
70
80
90
0 32 64 96 128
Eigenvectors
Householder
Backtransform
Total
GA Interface to PeIGS 3.0 (Solution of real symmetric generalized and standard
eigensystem problems)
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
Why NWChem Was Developed
Developed as part of the construction of the Environmental Molecular Sciences Laboratory (EMSL)
Envisioned to be used as an integrated component in solving DOE’s Grand Challenge environmental restoration problems
Designed and developed to be a highly efficient and portable MPP computational chemistry package
Provides computational chemistry solutions that are scalable with respect to chemical system size as well as MPP hardware size
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
New Theory Development
Theory (Method Development)
Algorithm Development
Preliminary Design Phase
Prototyping (Implementation)
Requirements Definition and Analysis Phase
Computational Chemistry Models
Detailed Design Phase
Implementation Phase
Acceptance Test Phase
Research including work in:
• Theory innovation
• O(N) reduction
• Improved MPP algorithms
• etc
NWChem Software Development Cycle
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
NWChem ArchitectureR
un-
tim
e da
taba
se
DFT energy, gradient, …
MD, NMR, Solvation, …
Optimize, Dynamics, …
SCF energy, gradient, …In
tegr
al A
PI
Geo
met
ry O
bje
ct
Bas
is S
et O
bje
ct
...PeI
GS
...
Global Arrays
Memory Allocator
Parallel IO
MolecularModelingToolkit
MolecularCalculation
Modules
MolecularSoftware
DevelopmentToolkit
GenericTasksEnergy, structure, …
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
• All major functions exist as independent modules.
• Modules only communicate thru the database and files
• No shared common blocks• The only argument to a
module is the database• Modules have well defined
actions• Modules can call other
modules
Program ModulesInput
SCF
DFT
MP2
CCSD
Stepper
Driver
nwArgos
Basis
Geometry
Energy
Filenames
Status
Database
Theory
Operation
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
NWChem is supported on at least these platforms, and is readily ported to essentially
all sequential and parallel computers. IBM SP
IBM workstations
CRAY T3
SGI SMP systems
Fujitsu VX/VPP
SUN and other Homogeneous workstation networks
x86-based workstations running Linux including laptops
x86-based workstations running NT or Win98
Tru64 and Linux Alpha servers (including SC series)
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
0
20
40
60
80
100
120
0 20 40 60 80 100 120
Speedup
LinearCRAY T3E-900
IBM SP2
iPSC/860 (Charmm Brooks et al.)
Myoglobin in Water 10,914 atoms
Rc=1.6 nm
0.36 / 108
0.94 / 128
0.17 / 108
0
250
500
750
1000
0 250 500 750 1000
Speedup
Number of nodes
LinearCRAY T3E-900
Octanol216,000 atoms
Rc=2.4 nm
0.58 / 1000
0
50
100
150
200
250
0 50 100 150 200 250
Speedup
Number of nodes
Linear
IBM SP2CRAY T3E-900
Dichloroethane-Water Interface 100,369 atoms
Rc=1.8 nm
1.0 / 250
1.6 / 250
MD Benchmarks
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
0
20
40
60
80
100
120
0 20 40 60 80 100 120
Speedup
LinearCRAY T3E-900
IBM SP2
iPSC/860 (Charmm Brooks et al.)
0.36 / 108
0.94 / 128
0.17 / 108
Wall clock times and scaling obtained for a 1000 step NWChem classical molecular dynamics simulation of myoglobin in water, using the AMBER force field, and a 1.6 nm cutoff radius. The system size is 10,914 atoms.
Myoglobin
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
GA sample program
status = ga_create(mt_dbl,n,n,'matrix',n,0,g_a) status = ga_create(mt_dbl,n,n,'matrix',n,0,g_b) status = ga_create(mt_dbl,n,n,'matrix',n,0,g_c) ...call ga_dgemm('N','N',n,n,n,1d0,g_a,g_b,0d0,g_c)
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
The SCF Algorithm in a Nutshell
( ) ( ) ( )
( ) ( ) [ ( )]
r r rD
F r r F r dr
; S[ S[ S [_ _
_ _
Density
Fock matrix
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
Acknowledgments
Jarek Nieplocha
The NWChem developers team
Environmental Molecular Sciences Laboratory
Pacific Northwest National LaboratorySPSCICOMP2000 8/15/2000 EA
NWChem DevelopersHPCC Developers
Staff
• Dr. Edo Apra
• Dr. Eric Bylaska
• Dr. Michel Dupuis
• Dr. George Fann
• Dr. Robert Harrison
• Dr. Rick Kendall
• Dr. Jeff Nichols
• Dr. T. P. Straatsma
• Dr. Theresa Windus
Research Fellows
• Dr. Ken Dyall
• Prof. Eric Glendenning
• Dr. Benny Johnson
• Prof. Joop van Lenthe
• Dr. Krzyzstof Wolinski
Post-Doctoral Fellows• Dr. James Anchell• Dr. Dave Bernholdt• Dr. Piotr Borowski• Dr. Terry Clark• Dr. Holger Dachsel• Dr. Miles Deegan• Dr. Bert de Jong• Dr. Herbert Fruchtl• Dr. Ramzi Kutteh• Dr. Xiping Long• Dr. Baoqi Meng• Dr. Gianni Sandrone• Dr. Mark Stave• Dr. Hugh Taylor• Dr. Adrian Wong• Dr. Zhiyong Zhang
Non-HPCC Developers
Staff• Dr. Dave Elwood• Dr. Maciej Gutowski• Dr. Anthony Hess• Dr. John Jaffe • Dr. Rik Littlefield• Dr. Jarek Nieplocha• Dr. Matt Rosing• Dr. Greg Thomas
Post-Doctoral Fellows• Dr. Zijing Lin• Dr. Rika Kobayash• Dr. Jialin Ju