IBM India © 2007 IBM Corporation http://w3.ibm.com/ibm/presentations Compiling and Executing Applications on BlueGene Subba R Bodda and Ravi K Komanduri IBM India Systems and Technology Laboratory
Feb 05, 2016
IBM India
© 2007 IBM Corporation
Compiling and Executing Applications on BlueGene
Subba R Bodda and Ravi K Komanduri
IBM India Systems and Technology Laboratory
2
IBM India
© 2007 IBM Corporation
Agenda
BlueGene setup
Compiling applications on BlueGene
Executing applications on BlueGene
3
IBM India
© 2007 IBM Corporation
Redbooks
Link: http://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query=blue+AND+gene
Blue Gene System Administration, SG24-7417-00 IBM System Blue Gene Solution: System Administration, SG24-7178-04 IBM System Blue Gene Solution: Configuring and Maintaining Your Environment, SG24-7352-00 Blue Gene/L: Hardware Overview and Planning, SG24-6796-02 IBM System Blue Gene Solution: Hardware Installation and Serviceability, SG24-6743-00 Blue Gene Safety Considerations, REDP-4257-00 Blue Gene/L: Safety Considerations, REDP-3983-01 Blue Gene/L: Performance Analysis Tools, SG24-7278-00 IBM System Blue Gene Solution: Application Development, SG24-7179-04 Unfolding the IBM eServer Blue Gene Solution, SG24-6686-00 GPFS Multicluster with the IBM System Blue Gene Solution and eHPS Clusters, REDP-4168-00 IBM System Blue Gene Solution Problem Determination Guide, SG24-7211-00 Evolution of the IBM System Blue Gene Solution, REDP-4247-00
4
IBM India
© 2007 IBM Corporation
BlueGene setup
bglfen01(10.16.28.30),bglfen02 is the front end node bglsn is the service node Login to bglfen and compile your code Use mpirun or LL to submit your jobs bglsn has the service daemons which would spawn your jobs on actual BG nodes
5
IBM India
© 2007 IBM Corporation
IBM Compilers for Blue Gene
IBM compilers (Fortran, C, C++) for Blue Gene are located on the frontend node in /opt/ibmcmp
Fortran:
/opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf
/opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf90
/opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf95 C:
/opt/ibmcmp/vac/bg/8.0/bin/blrts_xlc C++:
/opt/ibmcmp/vacpp/bg/8.0/bin/blrts_xlC
Note : Users can add the respective directories to their PATH.
6
IBM India
© 2007 IBM Corporation
GNU Compilers for Blue Gene
GNU compilers (Fortran, C, C++) for Blue Gene are located in /opt/blrts-gnu/
Fortran:
/opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-g77
C:
/opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-gcc
C++:
/opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-g++
7
IBM India
© 2007 IBM Corporation
MPI on Blue Gene
MPI implementation on Blue Gene is based on MPICH-2 from Argonne
National Laboratory.
Include files mpi.h and mpif.h are at the location:
-I/bgl/BlueLight/ppcfloor/bglsys/include
Libraries to link for MPI are:
-L/bgl/BlueLight/ppcfloorbglsys/lib -lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts
Note : The linking order for the MPI libraries should be as mentioned above.
8
IBM India
© 2007 IBM Corporation
Compiler Optimizations
9
IBM India
© 2007 IBM Corporation
Compiler Optimizations
10
IBM India
© 2007 IBM Corporation
IBM ESSL libraries
The IBM ESSL (Engineering Scientific Subroutine Library) is located in
/opt/ibmmath/essl/4.2/
11
IBM India
© 2007 IBM Corporation
Executing jobs using mpirun
cd /gpfs/fcHome/scratch/userid
mpirun –np 32 –partition R000 -cwd /gpfs/fcHome/scratch/ -exe /gpfs/fcHome/scratch/hello_c
Common options:
-args : List of arguments to the executables in double quotes
-env : List of environment variables in double quotes. “VARIABLE=value”
-mode : CO or VN (coprocessor/virtual-node)
-only_test_protocol : This options is used only to test communication protocol coprocessor mode : one MPI process per node, 1 GB limit per process.
Default mode virtual-node mode : two MPI processes per node, 512 MB limit per process; L3
cache, memory, networks, are shared
For more details type the following at a command prompt : mpirun -h
Note : A user can have only one job per Blue Gene partition
12
IBM India
© 2007 IBM Corporation
Executing jobs using submit_job Login to bglsn cd /bgl/BlueLight/ppcfloor/bglsys/bin ./mmcs_db_console mmcs$ allocate M03B_J214_64 mmcs$ submitjob M03B_J214_64 /bgl/hello.rts /bgl/ OK jobId=25 mmcs$ waitjob M03B_J214_64 25 OK mmcs$ free M03B_J214_64 OK mmcs$ quit
13
IBM India
© 2007 IBM Corporation
Environment Variables Environment variables can be specified using –env “XYZ=123 ABC=456”
Some of the environment variables are BGLMPI_EAGER BGL_APP_L1_WRITE_THROUGH BGL_APP_L1_SWOA BGLMPI_COPRO_SENDS BGLMPI_INTERRUPT
14
IBM India
© 2007 IBM Corporation
Redbooks
Link: http://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query=blue+AND+gene
Blue Gene System Administration, SG24-7417-00 IBM System Blue Gene Solution: System Administration, SG24-7178-04 IBM System Blue Gene Solution: Configuring and Maintaining Your Environment, SG24-7352-00 Blue Gene/L: Hardware Overview and Planning, SG24-6796-02 IBM System Blue Gene Solution: Hardware Installation and Serviceability, SG24-6743-00 Blue Gene Safety Considerations, REDP-4257-00 Blue Gene/L: Safety Considerations, REDP-3983-01 Blue Gene/L: Performance Analysis Tools, SG24-7278-00 IBM System Blue Gene Solution: Application Development, SG24-7179-04 Unfolding the IBM eServer Blue Gene Solution, SG24-6686-00 GPFS Multicluster with the IBM System Blue Gene Solution and eHPS Clusters, REDP-4168-00 IBM System Blue Gene Solution Problem Determination Guide, SG24-7211-00 Evolution of the IBM System Blue Gene Solution, REDP-4247-00
15
IBM India
© 2007 IBM Corporation
Summary
Compilation IBM XLF/XLC and GNU Compilers ESSL Libraries submit_job mpirun LL Environment Variables