Datasaabs vänner 15 April 2011 · 320 3.52 Intel E5462 HP DL160g5 2008 Q3 Mozart 64 0.2 Intel Itanium2 2006 Q3 17656 166.73 Peak Perf. [TFlops] Neolith Bore/Gimle Byvind ... 37 Dell

Niclas Andersson, [email protected] Supercomputer Centre, Sweden

Datasaabs vänner15 April 2011

National Supercomputer Centerin Sweden

● Provider of leading edge supercomputing resources to NSC partners SMHI and SAAB and to members of academic institutions troughout Sweden.

● The SNIC*-center at Linköping University

● Independent organisation within Linköping University

● Staff of 23 people

● Created 1989 when Linköping University purchased a Cray XMP in collaboration with SAAB.

*) Swedish National Infrastructure for Computing

Mission

● Design, procure and install and maintain computing and storage resources at large scale.

● Provide users with help, training and support to use supercomputers in the most efficient way.

● Manage, monitor, and coordinate systems, facilities, security, and usage.

● Develop and improve large scale computing

Major Partners and Funding Organisations

Meta-center for six supercomputer centers in Sweden: NSC, PDC, HPC2N, UPPMAX, C3SE, LUNARC

Provide funding for hardware and support personel for academic projects

Hosted by the Swedish Science Council NSCPDC

HPC2N

LUNARC

UPPMAX

C3SE

Swedish Meteorological and Hydrological InstituteManage and develop information on weather, water and climate that provides knowledge and advanced decision-making data for public services, the private sector and the general public.

Swedish Aeroplane ABProducts, services and solutions from military defence to civil security.

Swedish National Infrastructure for Computing

www.snic.vr.se

www.smhi.se

www.saabgroup.se

Computing ResourcesAcademic Projects (SNIC)

Neolith & KappaGeneral purpose clusters for academic projects in many different science fields: quantum physics, quantum chemistry, molecular dynamics, climate research, fluid dynamics, and many more.

MatterDedicated cluster for research on new materials for clean energy production, energy storage, emission reduction and nuclear waste disposal.

Neolith

Kappa

Matter

#180

#386

Matter Compute Cluster

● 512 compute servers: HP SL6000– 2 x E5520 (8 cores, 2.26 GHz)– 36 GiB memory– 500 GB disk

● 4 compute servers: HP DL160g6– 2 x X5570 (8 cores, 2.93 GHz)– 144 GiB memory– 4 TB disk

● 1 login server

● 1 system server for central services

Matter Networks

● Application network– QDR Infiniband: Voltaire– 40 Gb/s– Two level fat tree, 1:8 fan-out

● Storage and other services– 1 Gb/s to servers– 10 Gb/s backbone and to storage

● Lights-out network– 100 Mb/s– Monitoring and power control

Matter Power & Cooling

● 71 x 16A fuses● 140 kW sustained (avg)

● 5 Modular Cooling Systems (MCS)● Encapsulated racks (two-by-two)● In-row heat-exchangers● Computer inlet: 25 oC, outlet: 40 oC

Computing ResourcesSMHI

Gimle – Cluster for weather model development and climate research at SMHI

Byvind & Bore – Redundant pair of clusters for national weather forecast production at SMHI, including resources for emergencies.

Byvind, located at SMHI

Gimle & Bore

SMHI – Swedish Meteorological and Hydrological Institute (www.smhi.se)

Computing ResourcesSmaller systems

Smokerings – SNIC Grid computing cluster for Swegrid and WLCG

Skylord – SAAB cluster for simulations of aircrafts and UAVs: fluid dynamics, structural mechanics, and electro-magnetics. Located at SAAB.

Mozart – SMP with 512 GB of memory for various local projects with large memory requirements.

SAAB – Swedish Aeroplane AB (www.saabgroup.com) SNIC – Swedish National Infrastructure for Computing (www.snic.vr.se)

Computing Resources at NSCsummary

Name Cores Processor Server model Installed

[#]

6440 60.02 Intel E5345 HP DL140g3 2007 Q3Matter 4128 37.32 Intel E5520 HP SL6000 4in2U 2010 Q2Kappa 2912 26.32 Intel E5520 HP DL1000 4in2U 2009 Q4

1120 12.54 Intel E5462 HP DL160g5 2008 Q11024 9.27 Intel E5520 HP DL1000 4in2U 2009 Q4

1120 11.92 Intel X5550 HP SL6000 4in2U 2010 Q2

528 5.62 Intel E5430 2008 Q1

320 3.52 Intel E5462 HP DL160g5 2008 Q3

Mozart 64 0.2 Intel Itanium2 2006 Q317656 166.73

Peak Perf.

[TFlops]

Neolith

Bore/Gimle

Byvind

Smokerings Supermicro 2in1U

Skylord

SGI Altix 3700

● All HP systems are equipped with DDR or QDR Infiniband (Voltaire or Cisco)● Matter and Kappa are using HP MCS-rack for cooling● Neolith is #180 (47.03 TF) and Matter is #386 (34.28 TF) on Top500 (nov 2010)● Intel codenames: 53xx = Clovertown, 54xx = Harpertown, 55xx = Nehalem

Disk Storage

Accumulus – 1.4 PB

● Block storage for weather model development and climate research

● Lustre with access from Gimle

● 45 HP DL185g5, 37 Dell R510

Vagn – 0.77 PB

● Analysis resource and project storage for climate research and turbulence modelling

● GPFS integrated with analysis servers

● 7 Sand 5420 (Supermicro), 9 Xyratex 5404E

Disk Storagecont.

SNIC Central Storage – 0.62 PB

● Block storage for academic projects

● GPFS with access from Neolith, Kappa, and Matter (approx 1700 compute servers)

● 16 HP servers (DL380g5 & DL180g6),11 Nexsan SataBeast,1 IBM DS4700

Serenity – 1.3 PB

● Grid storage for SweStore and WLCG (part of the Nordic Tier 1)

● Dcache using SRM or GridFTP for access

● 8 HP DL360, 6 Sun X4540, 37 Dell R510

Disk Storagesummary

Name Type/FS Raw size[PB]1.40

Serenity 1.30GPFS 0.77

SNIC Central Storage GPFS 0.624.09

Accumulus LustreDcache

Vagn

Totalt [Petabyte]

● RAID-6 (two redundant disks) on 8 – 14 disk stripes.

Tape Storage

Bandida

● 3 servers

● IBM 3500 Tape robot

● 2881 slots in 5 frame

● 2050 tapes, 1.64 Petabyte

● 9 (+ 4) Tape stations, LTO 5, 4 (3)

● Backup (IBM TSM)

● Hierachical Storage Manager (HSM)

● Tape backend to Serenity (Dcache)

● Meteorological Archive and Retrieval System (MARS) för SMHI research, development, and forecast results.

Network Resources

Connectivity

● 2 x 10 Gb to LiU core routers

● 1 x 10 Gb to LHCOPN

● 2 x 1 Gb over LiU-net (lamda) to Campus Norrköping and further to SMHI (local network)

● 2 x 1 Gb to SMHI

Equipment

● 2 Cisco Catalyst 4506

● 2 Cisco Catalyst 3750e at SMHI

● HP ProCruve Switches

Computer Room Facilities

● Two computer rooms:● Hangaren (2007/2009) with district cooling ● Bunkern (2003) with chiller and filtered outside air

● Floorspace for IT-equipment: ca 360 m2

● Total used power currently: 830 kW (24/7/365) = 7.27 GWh/year

● ca 40% increase per year during the last 12 years● PUE: 1.1 – 1.3

● Reuse of energy?PUE=

Total facility powerIT equipment power

Cooling in Hangaren

Water dist.in comp. room

Open air coolingwithin computerroom

Dis

tric

t C

oo

ling

Dis

trib

uti

on

Computer room

Campus

Encapsulated Racks

Fans

Fans Pumps

10o/20o

15o/27o

14o/24o25o/40o

In/OutoC

A/W

A/WW/W

Tekniska Verken

OutsideAir

W/A

AbsorptionChillers

DistrictHeating

Building

System and Development Software

Own software stack on all resources

● Linux, CentOS distribution

● Open Fabric Enterprise Distribution (OFED)

● SLURM job management and job scheduling

● Intel Cluster Studio: compilers, debugger, IntelMPI, MKL, Trace analyzer and collector

● In-house developed software for installing, booting, monitoring, accounting, and management built on Python, Django, Nagios, BitTorrent, collectl, and many other packages.

Application Software

Quantum Chemistry

● Gaussian, NWChem, Dalton

Material Science

● VASP, Siesta, Wien2k, GPAW

Molecular Dynamics

● Gromacs, Desmond, Lammps

Bioinformatics

● BLAST, ClustalW, HMMER

Computational Fluid Dynamics

● Fluent, Edge, OpenFOAM

Climate● EC-Earth, IFS, Nemo,

RCA, RCO, Hiromb, Match, OASIS, ...

Numerical Weather Prediction

● Hirlam → Harmonie

Large Hadron Collider (LHC)

Geneva airport

LHC27 km circumference

100 meters underground

LHC Experiments

LHC Experiment: ATLAS

ATLAS Detector

IceCube Neutrino Observatory

● Detector at the South Pole

● 5160 photo multiplicators in 1 km3 ice

● Detect Čerenkov radiation from neutrinos → muons

Climate Simulation

Numerical Weather Prediction

Hurricane Katarina, 2005

Gudrun (Erwin), 2005, Byholma Timber storage

Challenges

● Stochastic process

● Chaotic nature of fluid dynamic equations

● Predict extreme weather conditions

● Increase in precision and accuracy

● Deadlines

Regional Weather Prediction