High-Performance Scientific Computing...we use the orthogonality properties of the Hermite polynomi-als: j Avgm = E[j Avg(!) m( (!))] E[m ( !))2], (6) where E[·] is the mathematical

High-Performance Scientific Computing

INFO 0939, Fall 2019

http://www.montefiore.ulg.ac.be/~geuzaine/INFO0939/

Based in part on material from Victor Eijkhout’s SSC 335/394 course at the Texas Advanced Computing Center

�1

Introduction

�2

Mathematics & Science• In science, we use mathematics to understand

physical systems.

• Different fields of science explore different ‘domains’ of the universe, and have their own sets of equations, encapsulated in theories.

• Determining the theories and governing equations requires observation or experimentation, and testing hypotheses.

�3

[Courtesy of San Diego Supercomputer Center]

�4

Scientific Computing• Why should we care about scientific computing?

– Computational research has emerged to complement experimental methods in basic research, design, optimization, and discovery in all facets of engineering and science

– In certain cases, computational simulations are the only possible approach to analyze a problem:• Experiments may be cost prohibitive (eg. flight testing a 1,000 fuselage/

wing-body configurations for a modern fighter aircraft)• Experiments may be impossible (eg. interaction effects between the

International Space Station and Shuttle during docking)– Simulation capabilities rely heavily on the underlying compute

power (e.g. amount of memory, total compute processors, and processor performance)• Fostered the introduction and development of super-computers starting

in the 1960’s• Large-scale compute power is tracked around the world via the Top500

List (more on that later)

�5

Scientific Computing: a definition

• “The efficient computation of constructive methods in applied mathematics”– Applied math: getting results out of application

areas– Numerical analysis: results need to be

correctly and efficiently computable– Computing: the algorithms need to be

implemented on modern hardware

�6

Examples of Scientific Computing (it really is everywhere)

Automotive

Streamlines for workstation ventilation

Heating, ventilation, and air conditioning

�7

Electrical Engineering

Examples of Scientific Computing (it really is everywhere)

F18 Store Separation

Aerospace

Weather Forecasting

�8

Examples of Scientific Computing (it really is everywhere)

�9

Temperature and natural convection currents in the eye following laser heating.

BiomedicalCOMPUMAG–SYDNEY 2011, STATIC FIELDS AND QUASI-STATIC FIELDS (III), PA10.2, CMP–305 2

Fig. 1. Mesh used for the numerical experiments [20]: a) full head (300,000 nodes, 27 tissues), b) grey matter, c) white matter.

matter �G(!) are modeled within a probabilistic framework,as functions of the random variable !. Therefore jAvg , jmax,j99�perc, eAvg , emax and e99�perc are random as well. Inparticular, by using the maximum entropy principle [22] wemodel (arbitrarily) �G(!) and �W (!) as independent randomvariables, uniformly distributed:

�G(!) ⇠ U([0.0753 ; 0.5155]) (S/m) (1)�W (!) ⇠ U([0.0533 ; 0.3020]) (S/m) (2)

C. The non intrusive approach

As the conductivities of the brain and the cerebellum aretwo independent random variables of finite variance, we canexpand them as a truncated series of order pin in the bi-dimensional Hermite polynomials of a random gaussian vector⇠(!) = (⇠1(!), ⇠2(!)), known as Hermite chaos polynomials

[18]:

�G(!) ⇡PinX

i=0

�Gi i(⇠(!)) (3)

�W (!) ⇡PinX

i=0

�Wi i(⇠(!)) (4)

where �Gi and �Wi are scalar values that depend on theprobabilistic law of the conductivities, Pin = Cpin

2+pinis the

number of bi-dimensional polynomials of order less than pin,and i is the ith bi-dimensional Hermite polynomial. Tosolve the stochastic problem, we use an approach based ona polynomial chaos decomposition of both the conductivityand the induced fields [18]. We assume the conductivities tobe of finite variance, with no assumption on the shape of theprobabilistic distribution.

The values of the induced fields—the average currentdensity in the brain jAvg(!) = jAvg(⇠(!))—are computedby the finite element method from any couple of values(�G(⇠(!)),�W (⇠(!))). The average density belongs to aspace that can be spanned by the polynomials (⇠(!)) andthus written as a truncated series to an order pout:

jAvg(!) =PoutX

m=0

jAvgm m(⇠(!)). (5)

To compute the value of the unknown real coefficients jAvgm,we use the orthogonality properties of the Hermite polynomi-als:

jAvgm =E[jAvg(!) m(⇠(!))]

E[ m(⇠(!))2], (6)

where E[·] is the mathematical expectation. The denominatorcan be computed analytically. The integral in the numeratoris computed by means of a Hermite Gauss integration schemewith d integration points [18]:

E[jAvg(!) m(⇠(!))] ⇡dX

i=1

...dX

j=1

wi,j(jAvg((t1, t2)i,j)) m((t1, t2)i,j), (7)

with (t1, t2)i,j the i, j-th Gauss point and wi,j the associatedweight in the bi-dimensional Cartesian rule. The deterministicproblem must thus be computed d2 times, with the conductiv-ity evaluated through (3) and (⇠1(!), ⇠2(!)) = (t1,= t2)i,j ,i, j = 1, . . . , d.

III. RESULTS AND DISCUSSION

The non intrusive method is governed by three parame-ters: pin, pout and d; pin is linked to the precision on the

Fields induced in human body close to power lines

�10

New Kinds of Computations

The Top500 List• http://www.top500.org• Owner submitted benchmark performance

sine 1993– based on a dense linear system solve– http://www.netlib.org/benchmark/hpl/

�11

�12

�13

�14

�15

�16

HPC and CECI• CECI is the “Consortium des

Equipements de Calcul Intensif” in Wallonia/Brussels– http://www.ceci-hpc.be– Once you create an account you

can use all the CECI clusters– Funded by FNRS– UCL, ULB, UNamur, UMons,

ULiège– Single login for all clusters (more

on that later)

�17

• Machines in CECI grid:– NIC4 @ ULiège– HMEM & Lemaitre2 @ UCL– Vega @ ULB– Hercules @ FUNDP– Dragon1 @ UMons

– 120 compute nodes with two 8-cores Intel E5-2650 processors at 2.0 GHz and 64 GB of RAM (4 GB/core)

– QDR Infiniband network– 144 TB FHGFS parallel

filesystem– The cluster is especially

designed for massively parallel jobs (MPI, several dozens of cores) with many communications and/or a lot of parallel disk I/O, 2 days max.

– SSH to nic4.segi.ulg.ac.be with the appropriate login and id_rsa.ceci file (more on that later).

�18

NIC4 System Summary

�19

External Power and Cooling

Class Goals/Topics• Remember that definition “The efficient

computation of constructive methods in applied mathematics”– Numerical analysis/algorithms, (parallel)

computation, and how to combine them• Theory topics: architecture, numerical

analysis, implementing the one on the other

• Practical skills: the tools of scientific computing

�20

Class Goals/Topics• UNIX Exposure

– shells/command line– environment– compilers– libraries

• Ideally: good practices for scientific software engineering– version control– build systems– data storage– debugging skills

�21

Class Setup• Theory classes on Tuesday @ 2pm• 2 or 3 projects, combining theory and

programming

�22

Computer Accounts• CECI clusters

– You should create an account now, and run your codes over there– Must access page from the ULiège subnet: https://login.ceci-hpc.be

• All required info is available on http://www.ceci-hpc.be• Don’t forget to read the FAQ!• Jobs run in a managed environment

– login to the login node– submit jobs to the scheduler– wait– collect results

• Production runs on the login node are forbidden– avoid resource intensive tasks– exceptions include compilers, “standard” UNIX commands (ls,

mkdir, etc.) �23

Remote Login on CECI Clusters• Only SSH access is allowed• All required info is available on – http://www.ceci-hpc.be

• Don’t forget to read the FAQ: – https://login.ceci-hpc.be/static/faq.html

• Cf. David Colignon's introduction later this afternoon

�24