Dr Mark Parsons Commercial Director, EPCC [email protected] +44 131 650 5022 Advanced Engineering Simulation and its future at EPCC Mark Parsons,

Dr Mark ParsonsCommercial Director, EPCC

[email protected]+44 131 650 5022

Advanced Engineering Simulation and its

future at EPCC

Mark Parsons, EPCC

Outline

• Introduction to the EPCC Industry Hub

• Recent advanced engineering simulation projects

• Challenges facing simulation in the next decade

eDIKT 2010

EPCC in 2010

• Supercomputing Centre at The University of Edinburgh

• 20 years old

• 75 staff– highly experienced– wide range of skills

• Multidisciplinary

• Multi-funded– turnover £4.5 million– 95% from external sources– 50% of turnover involves industry

• Provides National HPC services–HECToR is hosted at Advanced Computing Facility

eDIKT 2010

EPCC Industry Hub

• Cycles, Software and Skills

• One stop shop for industrysupercomputing

• Unique range of services– Cycles – provision of compute and data resources– Software – provision of simulation and

bespoke applications to meet industry challenges– Skills – understanding industry problems and finding solutions

• Benefits– Scotland – supporting local industry – creating jobs and growth– Inward investment – attracting companies eg. Xilinx, Cray– International leadership – worldwide research collaborations– Creating an important facility where we work together with end-user

companies and leading HPC hardware and software vendors

eDIKT 2010

HPC Adopter Programme

• Too few companies benefit from modelling and simulation

• Big hurdles to overcome for the new HPC user– Even for high-tech companies

• HPC Adopter Programme is designed to tackle this– Initial small 2-4 month projects designed to introduce companies to

benefits of HPC– Initial access to HECToR and other HPC systems donated by EPCC– Companies pay for staff effort only

– For Scottish companies Scottish Enterprise may also support staff costs through State Aid to company

– To date SE have supported 3 HPC Adopter Programme pilots– Other companies have already benefitted from programme

• Long term virtuous circle of repeat business

eDIKT 2010

Deep Casing Tools – Turbocaser

• Deep Casing Tools is a privately owned

SME in Aberdeen.

• Company focus is on innovative design

combined with precision engineering that

enable development and manufacture of

the next generation of casing and

completion tools for the oil and gas

industry.

• Deep Casing Tools Turbocaser uses a

motor powered by drilling-mud to ream oil

wells prior to pipe installation.

• Important to understand and optimise the

performance of the multi-stage motor.

• Drilling mud is a non-Newtonian fluid with

Reynold's Numbers between 4,000 and

20,000 and densities typically between 1.1

and 1.4 g/cm 3.

• Very computationally expensive to

performa multi-stage design studies.

• Ideal application for HPC: project modelled

mud flow through various Turbocaser

turbine designs using HECToR and

OpenFOAM

eDIKT 2010

OpenFOAM-based design study

• Built mesh from existing Deep Casing Tools' AutoCAD designs

• Initial single stage simulations

• Multistage design study

OpenFOAM•Free, open source CFD software package

•Large user base across commercial and academic organisations.

•Solves complex fluid flows involving chemical reactions, turbulence and heat transfer, solid dynamics and electromagnetics.

•Already ported to HECToR, also runs on commodity resources

eDIKT 2010

Project outcome

• Impact for company– Validation of basic design

– Better understanding of product behaviour

– Optimisation of pressure drop versus torque across multistage turbine

– Support for future product design and development

• Impact for EPCC– Demonstration of value of HPC

to real world problem

– Increased staff experience of Computational Fluid Dynamics

– Extended use of OpenFOAM on complex problem

– Potential for further use of HECToR eg. rotating mesh

eDIKT 2010

Prospect – Wind & Wave Modelling

The Company:Prospect is an engineering design and analysis provider to the world energy industries. Headquartered in Aberdeen, Prospect was founded by in 1999 and has grown rapidly since to become part of a 300 strong Group with operations spanning six continents

The Problem:• Simulations exist of blades, turbine, wind on

tower and waves on tower.

• Combining these simulations, using different software packages, is very difficult

• Massive computational resources required to keep all simulations sychronised

• Ideal application for HPC: EPCC asked to couple simulations and run result on HECToR

© Copyright Davagh and licensed for reuse under this Creative Commons Licence

eDIKT 2010

HPC solution: Coupled Simulation

• Create polyhedral / tetrahedral mesh to represent physical objects

• Simulate action of waves, ocean current and wind simultaneously

• Understand combined complex stresses on structure

Tools:

Simulia-Abaqus Finite Element Analysis

StarCCM Computational Fluid Dynamics

To

wer

Waves

Current

Wind

eDIKT 2010

Prospect Summary

Impact for Company

• First full simulation of entire deep-sea wind turbine

• In-house capacity to run 1 variation per night – HECToR can run 500 variations simultaneously

• Understanding impact of design changes, highlights efficiency savings

Impact for EPCC

• Breakthroughs in coupled simulations – FEA and CFD

• Gained experience of two industry codes, ported to HPC platforms

• Potential to perform further work for client and software vendors

© Copyright Davagh and licensed for reuse under this Creative Commons Licence

eDIKT 2010

Challenges facing modelling & simulation

• We are at a complex juncture in the history of

supercomputing

• For the past 20 years supercomputing has “hitched a lift” on

the microprocessor revolution driven by the PC

• Hardware has been surprisingly stable

• EPCC in 1994 had the 512 processor Cray T3D system– 0.0768 TFlops peak

• EPCC in 2010 retired the 2,560 processor IBM HPCx system– 15.36 TFlops peak – 200 x faster but only 5 x more processors ...

• But now we have a problem ...

eDIKT 2010

Average job size in Europe today

eDIKT 2010

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Jug

en

e

Jub

l

Lo

uh

i

XC

5

Le

gio

n

HE

CT

oR

Pa

lu

Ma

reN

ost

rum

Sta

llo

Pla

tine

Ne

olit

h

HP

Cx

HL

RB

II

BC

X

Hu

yge

ns

ZA

HIR

mu

rska

.csc

.fi

hw

w S

X-8

Jum

p

HE

RA

Ga

lera

Mili

pe

ia

TN

C

>2048

513-2048

129-512

33-128

<32

Data from PRACE Project

Multicore and Moore’s Law

• In 2005 microprocessor clock speeds peaked around 4GHz

• Exponential increase in individual core performance has

ceased – in fact individual performance is reducing– Serial codes are now running slower on your new system

• Theoretical performance of multi-core microprocessors

continues to rise

• Moore’s Law is alive and well– Relates to number of transistors on an area of silicon not clock speed

• But – today’s microprocessors are more and more difficult to

use– This can only get worse in coming years

eDIKT 2010

Meeting user needs?

• Hardware is leaving many HPC users and codes behind

• Majority of codes scale to less than 512 cores– these will soon be desk-side systems

• Less than 10 codes in EU today will scale on capability

systems with 100,000+ cores– Soon HECToR will have 70,000+ cores– Germany’s Jugene system already has 294,912 cores

• Many industrial codes scale very poorly – some codes will

soon find a laptop processor a challenge!

• Much hope is pinned on accelerator technology– But this has its own set of parallelism and programming challenges

eDIKT 2010

The hardware software disconnect

• We have lived through a golden age of easy programmability and

relaxed parallelism

• In the future codes will have to be highly scalable– But many were designed in a different age

• Main parallel programming paradigms were settled on 15-20

years ago– Many codes are written in Fortran, C or C++– Most use either MPI or OpenMP– They use mathematical algorithms developed for 10s or 100s of

processors – not hundreds of thousands

• Exascale systems now being planned will have many millions of

cores

eDIKT 2010

Reconnecting

• We need to focus much more effort on applications

programming

• Rethinking the mathematics behind modelling on massively

parallel systems

• Redesigning and re-implementing simulation and modelling

codes

• Building new languages and methods of expressing and

harnessing parallelism

• Unless we do this the justification for buying larger and more

powerful systems will become increasingly false

eDIKT 2010

Final comments

• The software area is huge

• We have challenges in

• The next few years are going to be very exciting in HPC

eDIKT 2010

Questions?

eDIKT 2010