Top Banner
www.ci.anl.gov www.ci.uchicago.edu The Cray XE6 Beagle Beagle Team (beagle [email protected] ) Computation Institute University of Chicago & Argonne National Laboratory
68
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

The Cray XE6 Beagle

Beagle Team ([email protected])Computation Institute University of Chicago & Argonne National Laboratory

Page 2: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

2 Intro to Beagle – [email protected]

Outline

• Role of the Computation Institute• Introduction to high performance computing• Beagle hardware• Basics about the work environment • Data transfer using Globus Online• Use of the compilers (C, C++, and Fortran)• Launching and monitoring applications• Using Matlab on Beagle• Using Python on Beagle

Page 3: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

Role of the Computation Institute

Director: Ian Foster

http://www.ci.uchicago.edu/

Contact: [email protected]

Page 4: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

4 Intro to Beagle – [email protected]

Computation Institute

• Joint Argonne/Chicago institute, with ~100 Fellows (~50 UChicago faculty) and ~60 staff

• Primary goals: – Pursue new discoveries using multi-disciplinary collaborations

and computational methods– Develop new computational methods and paradigms required

to tackle these problems, and create the computational tools required for the effective application of advanced methods at the largest scales

– Educate the next generation of investigators in the advanced methods and platforms required for discovery

We want to provide expertise for an efficient

and innovative use of Beagle

in imaging

Page 5: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

5 Intro to Beagle – [email protected]

How the CI supports people who use Beagle

Catalyst ProgramHel

p Des

k

Supp

ort

Training & Outreach

Perfor

man

ce

Engi

neer

ing

• Startup assistance• User administration assistance• Job management services• Technical support

BeagleServices

• User campaign management• Assistance with planning, reporting• Collaboration within science domains• Beagle point of coordination

• Performance engineering• Application tuning• Data analytics• I/O tuning

• Workshops & seminars• Customized training programs• On-line content & user guides

• Beagle’s wiki*

• Beagle’s web page**

* http://www.ci.uchicago.edu/wiki/bin/view/Beagle/WebHome

** http://beagle.ci.uchicago.edu/

Page 6: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

What the Heck is Supercomputing?

Credit: Henry Neeman, DirectorOU Supercomputing Center for Education & Research

http://www.oscer.ou.edu/education.phpContact: [email protected]

Page 7: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

7 Intro to Beagle – [email protected]

Why Beagle?

Page 8: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

8 Intro to Beagle – [email protected]

What affects performance? Accessing data

Imaging examples:1) Too many images to fit on a local disk,

use of network disks2) Image too big to fit into memory,

extensive use of swap space3) Operations too big to fit into cache

Page 9: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

9 Intro to Beagle – [email protected]

What affects performance? Repetition

Imaging examples:1) Experiments (MRI images, CT images …)

can be analyzed at the same time (if computer is large enough)

2) Each image or image set in a database can be analyzed independently

3) Slices or sub-images can be processed at the same time

Page 10: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

10 Intro to Beagle – [email protected]

Imaging examples:1) If analyzing a single image is time

consuming (or images are large): slices or sub-images can be processed at the same time using different threads

2) If images are small: different threads can analyze different images

Page 11: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

11 Intro to Beagle – [email protected]

Imaging examples:1) Experiments (MRI images, CT images …)

can be analyzed at the same time (if computer is large enough)

2) Each image or image set in a database can be analyzed independently on a distributed system

Page 12: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

12 Intro to Beagle – [email protected]

Page 13: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

13 Intro to Beagle – [email protected]

Page 14: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

Beagle hardware

PI: Ian Foster

http://beagle.ci.uchicago.edu/Contact: [email protected]

Page 15: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

15 Intro to Beagle – [email protected]

Beagle: hardware overview

• Ranked 65th fastest machine in the world (June 2011)

• Cray XE6 system• 151 teraflops• 17,660 compute cores• 23 TB memory• 600 TB disk (450 TB ormatted)• Cray Gemini interconnect

Page 16: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

16 Intro to Beagle – [email protected]

Beagle “under the hood”

Page 17: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

17 Intro to Beagle – [email protected]

Compute nodes

Compute nodes• 2 AMD Opteron 6100 “Magny-Cours”

• 12-core (24 per node)• 2.1-GHz

• 32 GB RAM (8 GB per processor)• No disk on node (mounts DVS and

Lustre network filesystems)

• To know more:http://www.ci.uchicago.edu/wiki/bin/view/Beagle/SystemSpecs#Overview

Page 18: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

18 Intro to Beagle – [email protected]

Details about the Processors (sockets)

• Superscalar:• 3 Integer ALUs• 3 Floating point ALUs (can do 4 FP per cycle)

• Cache hierarchy: • Victim cache • 64KB L1 instruction cache• 64KB L1 data cache (latency 3 cycles)• 512KB L2 cache per processor core (latency of 9 cycles)• 12MB shared L3 cache (latency 45 cycles)

• To know more:

http://www.ci.uchicago.edu/wiki/bin/view/Beagle/SystemSpecs

Page 19: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

19 Intro to Beagle – [email protected]

Interconnect

Interconnect• Communication between compute nodes

and with service nodes• Gemini Interconnect

• 2 nodes per Gemini ASIC• 4 x 12-cores (48 per Gemini)• Gemini are arranged in a 3D torus

• Latency ~ 1 μs• 168 GB/s bandwidth of switching capacity

(20 GB injection per node)• Resilient design

• To know more:http://www.ci.uchicago.edu/wiki/bin/view/Beagle/SystemSpecs#Details_about_the_Interconnect

Page 20: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

Using Beagle

PI: Ian Foster

http://beagle.ci.uchicago.edu/Contact: [email protected]

Page 21: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

21 Intro to Beagle – [email protected]

Steps for computing on Beagle

• You need a user id on Beagle• You need an active project• You need to understand the basics of how the system

works (check files, move files, create directories)• You need to move your data to Beagle• The application(s) that perform the calculations need

to be installed on Beagle• You need to submit and monitor your jobs to the

compute nodes• You need to transfer your data back to your system

Page 22: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

22 Intro to Beagle – [email protected]

What you need to get started on Beagle

• A CI account: if you don’t have it, get one – https://accounts.ci.uchicago.edu/– You will need some person at the CI to sponsor you, this person can be:

o Your PI, if he or she is part of the CIo A collaborator that is part of the CIo A catalyst you will be working with

– You must upload an SSH public key to log in• A CI project (for accounting)

– https://www.ci.uchicago.edu/hpc/projects/o For joining an HPC projecto For creating a new HPC project

– This will change later this year, to let allocations committee make decisions• To know more about CI account and HPC basics

– http://www.ci.uchicago.edu/faq• To know more about Beagle accounts and basics

– http://www.ci.uchicago.edu/wiki/bin/view/Beagle/HowToStart

Page 23: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

23 Intro to Beagle – [email protected]

Beagle’s operating system

• Cray XE6 uses Cray Linux Environment v3 (CLE3) • SuSE Linux-based • Compute nodes use Compute Node Linux (CNL)• Login and sandbox nodes use a more standard Linux• The two are different.• Compute nodes can operate in

– ESM (extreme scalability mode) to optimize performance to large multi-node calculations

– CCM (cluster compatibility mode) for out-of-the-box compatibility with Linux/ x86 versions of software – without recompilation or relinking!

• To know more:http://www.ci.uchicago.edu/wiki/bin/view/Beagle/ComputeOnBeagle#Basics_about_the_work_environmen

Page 24: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

24 Intro to Beagle – [email protected]

Beagle’s filesystems

• /lustre/beagle: local Lustre filesystem (read-write) -- this is where images should be kept for computation and where output files should be written, BUT NO BACKUP!

• /gpfs/pads: PADS GPFS (read-write) – for storing permanently images and results

• /home: CI home directories, largely useless

• To know more:http://www.ci.uchicago.edu/wiki/bin/view/Beagle/ComputeOnBeagle#How_to_work_on_the_filesystem

Page 25: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

Handling data on Beagle

http://beagle.ci.uchicago.edu/Contact: [email protected]

Page 26: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

26 Intro to Beagle – [email protected]

How to move data to and from Beagle• Beagle is not HIPAA-compliant — no PHI data on Beagle • Example of factors for choosing a data movement tool:

– how many files, how large the files are …– how much fault tolerance is desired,– performance– security requirements, and – the overhead needed for software setup.

• Recommended tools:– scp/sftp can be OK for moving a few small files

o pros: quick to initiate o cons: slow and not scalable

– For optimal speed and reliability we recommend Globus Online :o high-performance (e.g., fast)o reliable and easy to useo easy to use from either a command line or web browser,o provides fault tolerant, fire-and-forget transfers. If you know you'll be moving a lot of data

or find scp is too slow/unreliable we recommend

• To know more:http://www.ci.uchicago.edu/wiki/bin/view/Beagle/ComputeOnBeagle#How_to_move_data_to_and_from_Bea

Page 27: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

27 Intro to Beagle – [email protected]

Trivial, right?

27

“I need my data over there – at my _____”

(supercomputing center, campus server, etc.)

Data Source

Data Destination

Getting data to the right place…

Page 28: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

28 Intro to Beagle – [email protected]

Reality: it is tedious and time-consuming

28

Data Source

Data Destination

“GAAAH!

%&@#&”

What’s the big deal?

Page 29: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu29

How It Works

DataSource

DataDestination

User initiates transfer request

1

Globus Online moves files

2

Globus Online notifies user

3

How It Works

Page 30: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu30

How It WorksGetting Started (2 easy steps)

1. Sign up: Visit www.globusonline.org to create an account

Page 31: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu31

How It WorksGetting Started (2 easy steps)

2. Start moving files: Pick your data and where you want to move it, then click to transfer

Page 32: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu32

How It WorksFile Movement Options

We strive to make Globus Online broadly accessible…

• You can just move files using the Web GUI

• To automate workflows you use the Command Line Interface (CLI)

• To know more: (quickstart, tutorials, FAQs …)https://www.globusonline.org/resources/

Page 33: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

33 Intro to Beagle – [email protected]

Steps for computing on Beagle

• You need a user id on Beagle• You need an active project• You need to understand the basics of how the system

works (check files, move files, create directories)• You need to move your data to Beagle• The application(s) that perform the calculations need

to be installed on Beagle• You need to submit and monitor your jobs to the

compute nodes• You need to transfer your data back to your system

✔✔

Page 34: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

Working with applications on Beagle

PI: Ian Foster

http://beagle.ci.uchicago.edu/Contact: [email protected]

Page 35: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

35 Intro to Beagle – [email protected]

Applications on Beagle

• Applications on Beagle are (mostly) run from the command line, e.g.:

aprun –n 17664 myMPIapp <myInput >& this.log

• How do I know if an application is on Beagle?– http://beagle.ci.uchicago.edu/software/– http://www.ci.uchicago.edu/wiki/bin/view/Beagle/SoftwareOnBeagle– On Beagle, use module avail, e.g.:lpesce@login2:~> module avail 2>&1 | grep –i matlab

Matlab/7.13(default)

• What if it isn’t there?• What if I want to use my own application?

Page 36: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

36 Intro to Beagle – [email protected]

In passing … Modules and work environment

• Modules sets the environment necessary to use a specific to applications, collection of applications, or libraries

• A module dynamically modifies the user environment• The module command provides a number of capabilities

including:– Rendering an application accessible: (module load)– unloading a module (module unload)– Changing a version of an application or an enviroment, i.e.,

unloading a module and loading another (module swap)– listing which modules are currently loaded (module list)– determining which modules are available and could be loaded

(module avail)

Page 37: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

37 Intro to Beagle – [email protected]

If you want a tool that isn’t on Beagle

For any specific requirements, submit a ticket to [email protected] the following information:• Research project, group and/or PI• Name(s) of software packages(s)• Intended use and/or purpose• Licensing requirements (if applicable)• Specific instructions or preferences (specific

release/version/vendor, associated packages, URLs for download, etc.)

Page 38: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

38 Intro to Beagle – [email protected]

If you want to port your own application

lpesce@login2:~> module avail PrgEnv

lpesce@login2:~> module avail 2>&1 | grep PrgEnv

PrgEnv-cray/1.0.2

PrgEnv-cray/3.1.49A

PrgEnv-cray/3.1.61(default)

PrgEnv-gnu/3.1.49A

PrgEnv-gnu/3.1.61(default)

PrgEnv-pgi/3.1.49A

PrgEnv-pgi/3.1.61(default)

Cray compilers

-Excellent Fortran

-CAF and UPC Gnu compilers

-Excellent C - Standard PGI compilers

-Excellent Fortran

-Reliable

You’ll need a compilerwhich is part of a programming

environment

Page 39: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

39 Intro to Beagle – [email protected]

User’s guides and man pages

• PGI: – http://www.pgroup.com/resources/docs.htm– Or type man pgf90, man pgcc, man pgCC

• GCC: – http://gcc.gnu.org/onlinedocs/– Or type man gfortran, man gcc, man g++

• Cray: under “Programming Environment”– http://docs.cray.com/cgi-bin/craydoc.cgi?mode=SiteMap;f=xe_sitemap– Or type man crayftn, man craycc, man crayc++

Page 40: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

40 Intro to Beagle – [email protected]

Compiling on Beagle: environment set up

• Native, optimized versions of many numerical libraries are available for the Cray XE6http://docs.cray.com/cgi-bin/craydoc.cgi?mode=SiteMap;f=xe_sitemapunder “Math and Science Libraries”

– For a list of all libraries installed on Beagle use:module avail 2>&1 | less

• Libraries are simple to use, via the module commande.g., FFTW, via module load fftw

Page 41: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

41 Intro to Beagle – [email protected]

Compiling on Beagle: libraries & docs

• AMD Core Math Library (ACML) manual• SuperLU Users' Guide• LAPACK man pages• ScaLAPACK man pages• BLACS and BLAS man pages• Fast_mv man pages• PETSc man pages• Trilinos man page• Introduction to Iterative Refinement Toolkit (IRT) man pages• Introduction to Cray LibSci FFT man pages• Introduction to FFTW 3.2.x man page• Introduction to FFTW 2.1.5 man page

Page 42: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

42 Intro to Beagle – [email protected]

More details about the environment

• Beagle can use both statically and dynamically linked (shared) libraries

http://www.ci.uchicago.edu/wiki/bin/view/Beagle/DevelopOnBeagle#Static_vs_Dynamic_linking

• All compilers on Beagle support:– MPI (Message Passing Interface, standard for

distributed computing) and– OpenMP (standard for shared memory computing).

Note: flags activating openMP pragmas or directives might be different among compilers, see man pages.

• Some compilers support also PGAS languages (e.g., CAF or UPC), for example the Cray compilers

Page 43: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

43 Intro to Beagle – [email protected]

Steps for computing on Beagle

• You need a user id on Beagle• You need an active project• You need to understand the basics of how the system

works (check files, move files, create directories)• You need to move your data to Beagle• The application(s) that perform the calculations need

to be installed on Beagle• You need to submit and monitor your jobs to the

compute nodes• You need to transfer your data back to your system

✔✔

Page 44: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

44 Intro to Beagle – [email protected]

On running jobs on compute nodes

• The system operates through a resource manager (Torque) and a scheduler (Moab) and it is based on PBS scripts

• Beagle CLE (Cray Linux Environment) supports both interactive and batch computations

• When running applications on the compute nodes, it is best to work from the login nodes (as opposed to the sandbox node, which is better used to develop)

• It is not possible to log in on the compute nodes

Page 45: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

45 Intro to Beagle – [email protected]

Launching an application on compute nodes

They are all usually part of a PBS (Portable Batch System) script:• The first step is to obtain resources which utilizes the qsub command

• The second step is to set the appropriate environment to run the calculations

• The third step is to move input files, personal libraries and applications to the Lustre file system

• The fourth step is to run the application on the compute nodes using the application launcher (aprun)

• The final step is to move files back to /home or /gpfs/pads/projects

Page 46: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

46 Intro to Beagle – [email protected]

First step: request resources with qsub

• Users cannot access compute nodes without a resource request managed by Torque/Moab

• That is, you will always need to use qsub• Typical calls to qsub are:

– For an interactive jobqsub -I -l walltime=00:10:00,mppwidth=24

– for a batch jobqsub my_script.pbs

Page 47: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

47 Intro to Beagle – [email protected]

Interactive

• When you run interactive jobs you will see a qsub prologue:

lpesce@login2:~> qsub -I –l walltime=00:10:00,mppwidth=24

qsub: waiting for job 190339.sdb to start

qsub: job 190339.sdb ready

############################# Beagle Job Start ##################

# Job ID: 190339 Project: CI-CCR000070 #

# Start time: Tue Jul 26 12:23:14 CDT 2011 #

# Resources: walltime=00:10:00 #

##############################################################

• After you receive a prompt, you can run your jobs via aprun:

aprun –n 24 myjob.exe <myinput >& my_log

lpesce@login2:~> aprun –n 24 myjob2.exe <myinput2 >& my_log2

lpesce@login2:~>

• Good for debugging and small tests• Limited to one node (24 cores)

Page 48: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

48 Intro to Beagle – [email protected]

Batch scripts

• Batch scheduling is usually done with a PBS script

• Scripts can be very complex• Note: the script is executed on the login node!

Only what follows the aprun command is run on the compute nodes

• We’ll look into simple scripts

• To know more:http://www.ci.uchicago.edu/wiki/bin/view/Beagle/ComputeOnBeagle#How_to_submit_jobs

Page 49: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

49 Intro to Beagle – [email protected]

Example of an MPI script!/bin/bash#PBS -N MyMPITest#PBS -l walltime=1:00:00#PBS -l mppwidth=240#PBS -j oe

#Move to the directory where the script was submitted -- by the qsub commandcd $PBS_O_WORKDIR# Define and create a directory on /lustre/beagle where to run the jobLUSTREDIR=/lustre/beagle/`whoami`/MyMPITest/${PBS_JOBID}echo $LUSTREDIRmkdir -p $LUSTREDIR

# Copy the input file and executable to /lustre/beagle cp /home/lpesce/tests/openMPTest/src/hello_smp hello.in $LUSTREDIR # Move to /lustre/beaglecd $LUSTREDIR

# Note that here I was running hello_smp on 240 cores , i.e., using 240 PEs (by using -n 240)# each with 1 thread -- i.e., just itself (default by not using -d)aprun -n 240 hello_smp <hello.in > hello.out3

• Set shell (I use bash)• Give a name to the job• Set wall time to 1 hr (hh:mm:ss)• Ask to merge err and output from the

scheduler • $PBS_O_WORKDIR: directory from where the script was submitted

• Name, output and make a directory on lustre

• Move all the files that will be used to lustre

• Go to lustre

• Use aprun to send the computation to the compute nodes

• -n 240 asks for 240 MPI processor elements (“cores,” essentially)

Page 50: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

50 Intro to Beagle – [email protected]

Example of an openMP script#!/bin/bash#PBS -N MyOMPTest#PBS -l walltime=48:00:00#PBS -l mppwidth=24#PBS -j oe

#Move to the directory where the script was submitted -- by the qsub commandcd $PBS_O_WORKDIR# Define and create a directory on /lustre/beagle where to run the jobLUSTREDIR=/lustre/beagle/`whoami`/MyTest/${PBS_JOBID}echo $LUSTREDIRmkdir -p $LUSTREDIR

# Copy the input file and executable to /lustre/beagle, these have to be user and project specificcp /home/lpesce/tests/openMPTest/src/hello_smp hello.in $LUSTREDIR # Move to /lustre/beaglecd $LUSTREDIR

# Note that here I was running one PE (by using -n 1)# each with 24 threads (by using -d 24)# Notice the setting of the environmental variable OMP_NUM_THREADS for openMP# if other multi-threading approaches are used they might need to be handled differently

OMP_NUM_THREADS=24 aprun -n 1 -d 24./hello_smp <hello.in > hello.out4

• Set shell (I use bash)• Give a name to the job• Set wall time to max: 48 hrs

(hh:mm:ss)• Ask to merge err and output from the

scheduler• $PBS_O_WORKDIR: directory

from where the script was submitted

• Name, output and make a directory on lustre

• Move all the files that will be used to lustre

• Go to lustre

• Use aprun to send the computation to the compute nodes

• First set environmental variable OMP_NUM_THREADS to desired value (24 is rarely optimal!)

• -d 24 asks for 24 OMP processes per MPI process

• -n 1 asks for only one MPI process

Page 51: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

51 Intro to Beagle – [email protected]

Queue Name

Max Walltime

Max # nodes

Default # nodes

Max # jobs in queue

Total # Reserved nodes

Interactive 4 hour 1 1 1 8

development 30 min 3 1 2 16

scalability 30 min 10 1 4 10

batch 2 days 740 1 none N/A• Recommended as second step, after the code compiles and runs using the

interactive queue on one node• To test parallelism on a small scale • Up to 3 nodes..• Provides dedicated resources to efficiently optimize and test parallelism

Default queue, to run all the rest

Recap of queues available on Beagle

• Recommended as first step in porting applications to Beagle• To test and debug code in real time. • On one node.• Provides dedicated resources to run continuous refinement sessions

• Recommended as third step after parallelism was tested on a small scale • Up to 10 nodes..• Provides dedicated resources to efficiently test and refine scalability

• To know more:http://www.ci.uchicago.edu/wiki/bin/view/Beagle/SchedulingPolicy

Page 52: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

52 Intro to Beagle – [email protected]

More about aprun

• The number of processors, both for MPI and openMP, is determined at launch time by the aprun command (more or less that is)

• The aprun application launcher handles stdin, stdout and strerr for the user’s application

• To know more:http://www.ci.uchicago.edu/wiki/bin/view/Beagle/ComputeOnBeagle#AprunOr type man aprun

Page 53: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

53 Intro to Beagle – [email protected]

To monitor applications and queues

• qsub batch jobs are submitted using the qsub command• qdel is used to delete a job• qstat shows the jobs the resource manager, Torque, knows

about (i.e., all those submitted using qsub). – qstat -a show all jobs in submit order– qstat -a -u username show all jobs of a specific user in submit

order– qstat -f job_id receive a detailed report on the job status– qstat -n job_id what nodes is a job running on– qstat -q gives the list of the queues available on Beagle

• showq show all jobs in priority order. showq tells which jobs Moab, the scheduler, is considering eligible to run or is running

• showres showres show all the reservations currently in place or that have been scheduled

• To know more:http://www.ci.uchicago.edu/wiki/bin/view/Beagle/ComputeOnBeagle#commands_for_submitting_and_inqu

Page 54: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

Applications on Beagle

http://www.ci.uchicago.edu/wiki/bin/view/Beagle/WebHome

Contact: [email protected]

Page 55: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

55 Intro to Beagle – [email protected]

Applications on Beagle

• GUIs are in general not supported• Licensing is similar to any other uchicago.edu

machine– Some vendors charge by number of cores, which can

be expensive on Beagle– Most applications which have a campus license can

be simply installed and used on Beagle• All software which is available at no chance can

in principle be installed on Beagle even if porting might not be trivial (e.g., Octave)

Page 56: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

Matlab

http://www.ci.uchicago.edu/wiki/bin/view/Beagle/MATLABhttp://beagle.ci.uchicago.edu/

Contact: [email protected]

Page 57: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

57 Intro to Beagle – [email protected]

Using Matlab on Beagle: basics

• The Matlab GUI is not supported and most likely will not be in the future:– According to our experience Matlab is not very effective in exploiting

massively parallel supercomputers such as Beagle– If you have reasons to believe otherwise speak up now.

• Compiled executables from Matlab code can be easily run on Beagle:– MATLAB programs should be compiled using mcc (Matlab compiler)

and run as command line executables with MCR (Matlab Compiler Runtime).

– In our experience, Matlab has shown very limited ability in exploiting effectively multi-core processors.

– Therefore, to exploit parallelism, the resulting executables, should be run in parallel using a scripting language such as a Bash script or a Swift script.

Page 58: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

58 Intro to Beagle – [email protected]

Using Matlab on Beagle: mcc and MCR

• The Matlab compiler produces executables that in order to run require the Matlab Compiler Runtime (MCR), a set of shared libraries that enables the execution of Matlab files without an installed version of Matlab.

• Currently MCR is available as /soft/matlab/7.13/ /soft/mcr/v714/ (if you require other versions let us know).

• The mcr compiler is loaded with the command module load matlab

• See alsohttp://www.mathworks.com/help/toolbox/compiler

Page 59: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

59 Intro to Beagle – [email protected]

Using Matlab on Beagle: Matlab code

• Any Matlab function of the form

foofunc(x1,x2,... ,xn) can be turned into an executable using the MATLAB compiler -- some black magic required, see http://www.ci.uchicago.edu/wiki/bin/view/Beagle/MATLAB#MATLAB_functions_that_can_be_com

• Matlab functions can call other Matlab functions from other files, usually leaving them in the compilation directory will be sufficient

• Matlab seems to have problems running multi-threaded programs efficiently, compile single-threaded, e.g.:

mcc -R -singleCompThread -R -nojvm -R -nodisplay -mv myapp.m -o my_app

Page 60: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

60 Intro to Beagle – [email protected]

Using MATLAB on Beagle: mcc output

• After the compilation, a number of files will be generated:– mccExcludedFiles.log : don’t worry about this one– my_app: the executable you will need to copy to Beagle– readme.txt : contains information, for example where is

the version of MCRInstaller.bin for your specific MATLAB, which you will need if different from the ones available on Beagle

– run_my_app.sh : a shell script that can is used to run each copy of my_app. We recommend that you use it to avoid having to take care of too many variables in your PBS scripts. However, you will need to modify those scripts when using them on Beagle (details on web site).

Page 61: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

61 Intro to Beagle – [email protected]

Using Matlab on Beagle: note

• Run multiple copies of single-threaded run_my_app.sh using a scripting language:– Bash shell– Swift

• In general Matlab compiled executables do not use Beagle very efficiently (both in terms of CPU and memory) and this should be considered carefully when planning large calculations.

• Let us know if we can help.

Page 62: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

Python

http://www.ci.uchicago.edu/wiki/bin/view/Beagle/PYTHONhttp://beagle.ci.uchicago.edu/

Contact: [email protected]

Page 63: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

63 Intro to Beagle – [email protected]

Using Python on Beagle: basics

• Python version 2.6 is the default.• Python version 2.7.1 is also available and can be used

instead ( by typing module load python). • Also Numpy and Scipy are available.• Pil is also available• Computations that involve inlined C code can use the Weave

module, which is part of Scipy• Computations that involve Fortran or C shared libraries can

use f2c, which is part of Scipy. These calculations appear to be capable of using OpenMP effectively (we did not try MPI).

• Other packages and versions can be installed if users request them.

Page 64: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

64 Intro to Beagle – [email protected]

Python on Beagle: Parallel shared memory

• We used the multiprocessing package because it side-steps the Global Interpreter Lock that renders the threading module unable to run multiple threads concurrently. (Can also run in distributed memory.)

• Semaphore and barrier functionalities appear to be working properly on Beagle

Page 65: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

65 Intro to Beagle – [email protected]

Python on Beagle: distributed memory

• The multiprocessing can also run in distributed memory, but we haven’t used it yet.

• We are planning to test global arrays bindings for Python

• Mpi4py should be installed and tested soon• Distributed memory was utilized so far by

running multiple instances of the same python code on different inputs on different nodes via a script

Page 66: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

Can you suggest other applications we should include?

Write to [email protected]

Page 67: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

67 Intro to Beagle – [email protected]

Acknowledgments

• BSD for funding most of the operational costs of Beagle• A lot of the images and the content has been taken or learned

from Cray documentation or their staff• Globus for providing us with many slides and support; special

thanks to Mary Bass, manager for communications and outreach at the CI.

• NERSC and its personnel provided us with both material and direct instruction; special thanks to Katie Antypas, group leader of the User Services Group at NERSC

• All the people at the CI who supported our work, from administrating the facilities to taking pictures of Beagle

• Beagle users who helped with the content about using Matlab and Python

Page 68: Www.ci.anl.gov  The Cray XE6 Beagle Beagle Team (beagle-support@ci.uchicago.edu)beagle-support@ci.uchicago.edu Computation Institute.

www.ci.anl.govwww.ci.uchicago.edu

Thanks!

We look forward to working with you.

Questions?

(or later: [email protected])