Introduction to Grid Computing with High Performance Computing.

Introduction to Grid Computing withHigh Performance Computing

• Objectives• Training Units• Introduction to Grid Computing• Registration with the White Rose Grid• Information Sources

Outline

Learning Outcomes

• Develop a High Performance computing (HPC) application.• Develop and manage applications for Computational grids• Manage the execution of HPC applications over a grid

computing system.• Discover resources, applications and data on distributed

systems.• Demonstrate an appreciation of evolving Grid

Technologies.

Sections

• Introduction to the White Rose Grid and grid operating systems.

• Application development for computational grids.

• Development tools and techniques for distributed high performance computing applications.

• Grid technologies and example grid projects

Assessment

• 3 Sets of assessment problems• A mini project on grid computing with a 3000

word report• Marks for the course are broken down as

follows– 40% from assessed problems– 60% from mini project

Grid Computing Mini Project

• Grid Applications for Specific Disciplines– Design of a grid application– Implementation of a prototype– An essay on the grid technologies that will influence

different disciplines and how this will be achieved.

Introduction to the White Rose Grid and Grid Operating Systems• Introduction to Grids, Registration and Access• Application management and development

using the UNIX operating system.• Review of White Rose Grid architecture and

applications.• Job Management and Scheduling.• Middleware for grid computing.

Application Development for Computational Grids

• Using and developing matlab applications for the grid.• Development of HPC applications using the C and

Fortran• Review of object orientation techniques in the

development of HPC applications.• Application Development using C++• Application development environments for the White

Rose Grid

Development tools and techniques for distributed high performance computing applications.• Libraries for HPC Application Development on

the WRG.• Performance Assessment Tools• Distributed Application Development Using MPI• Distributed Application Development Using

Open MP

Grid technologies and example grid projects• Introduction to Grid Services and the Open

Grid Services Architecture.• Development of Grid Portals• Implementing a project as a grid service, The

DAME project.• Cluster building and Integration with Grids• From semantic web to semantic grids

– Distributed searching of textual databases

Grid Computing is New Technology• Grid computing is in the

pioneering age!• As grids and grid

technology evolves it is to be expected that the contents of course material will change.

Introduction to Grid Computing

• Grid Technologies• Grid Definition• Types of Grid• Grid Architecture

Grid Technologies

• Large scale multi site data mining and data fusion• Shared virtual reality• Interactive collaboration• Real-time access to remote resources.

Grid Technologies

Access Grid

Definition of the Grid Problem

The “Grid Problem” is defined as the flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions and resources.

From “Anatomy of the Grid (Enabling Scalable Virtual

Organisations)”

I.Foster et al. Intl. J. Supercomputer Applications 2001.

Characteristics of Grids

•Global collections of resources connected with high speed networks

•Supercomputers, databases, storage, instruments, immersive environments

•Next generation internet•Emerging technology•Cost effective management of high end compute and data resources

Grid Architecture

Networks – High Bandwidth

ComputingTflops

Data storagePeta byte

TheGrid

Virtual Organisations

Evolution of Networked Computing

Architecture

Client Server

Distributed

Web Applications

Web Services

Grid Applications

Grid Services

Description

Network is computer

Objects The web Computer is Network

Virtual Organisations

Globally Extended Virtual Organisations

Peak Late 80’s

Early 90’s Late 90’s Early 2k Late 2k?? ??

Protocol X X HTTP, JVM XML, LDAPSOAPWSDL

GRIP, GRAM, GFTP,LDAP

MDS,UDDI

The Internet and Dynamic Web Technology

The Web Application Model

Web Services

Web service are OK but…

They can only run on the computerWhere they are put.

The computer or dataavailability may not be fit for the task.

Web Services Model

Types of Grid

Computational Grid Distributed Supercomputing

Grid aggregates computational resources to compute large complex problems

High throughput computing

Grid Schedules large numbers of loosely coupled or independent tasks

On Demand Computing Meet short term requirements for projects that cannot be cost-effectively or conveniently computed locally

Data Grid  Data Intensive Computing Grid Synthesises from data managed on geographically distributed repositories

Access Grid  Collaborative Computing Enhancing human-to-human interactions, e.g. collaborative design and virtual worlds

Grid Services

Web services onSteroids! They go anywhere!

Take on any task!

Grid Services

Grid ArchitectureAPPLICATION Molecular Modelling, Aerospace, Graphics and

Visualisation

APPLICATION TOOLKITS Data Grid, Remote Computation, Portals, Collaborative Visualisation, Sensors

GRID SERVICES Protocols, Authentication, Policy, Resource Management, Instrumentation

GRID FABRIC Storage, Computers, Networks, Sensors, Devices, Experiments

Grid Examples

•NASA’s Information Power Grid•computing infrastructure connecting the resources of several of its R&D Laboratoriesforming the Information Power Grid or IPG

•High Energy Physics E Data Grid

•The DataGrid Project is providing thesolution for storing and processing data generated by the LHC at CERN.

Grid and Web Service Demonstrations• SETI@home• The Dame Portal

– Requires valid e-Science Certificate

• Virtual Observatory• European Data Grid Demonstrators• OpenGIS Web Service• Matlab data Mining Service

Statistics for SETI at Home (16/10/2003)

Total Last 24 Hours

Users 4710399 1534

Results received 1073252223 1377476

Total CPU time 1655366.998 years

1347.953 years

Floating PointOperations

3.769869e+21 5.372156e+18(62.18 TeraFLOPs/sec)

Average CPU timeper work unit

13 hr 30 min 40.6 sec

8 hr 34 min 20.0 sec

SETI@home’s Most Promising Candidates

The DAME Portal

• Distributed Aircraft Maintenance Environment• Portal runs engine vibration analysis tools on

selected sets of engine performance data.• http://iri02.leeds.ac.uk:9080/damexto/

damexto

Data Grid

• Portal Demonstrators– Medical Imaging– Bioinformatics

• http://edg-wp10.healthgrid.org

Visit the Virtual Sky Portal

http://virtualsky.org

OpenGIS Web Service

• http://www.opengis.org/resources/?page=demos• Standards based• Data Connections

– Forestry information– Satellite data Landsat 7, ESA Global imagery data– Atlas data, Roads, boundaries, populations, water

areas..

• Based on Web services enables easy integration with OGSA based Grid Services

e-Science Projects

•Distributed Aircraft Maintenance Environment•Distributed Collaborative Visualisation and problem solving environments•Clinical and Biomedical projects

• A co-operative clinical e-Science Framework•Biosim GRID•Decision making in the health sector

Lots of projects Here are a few!!

White Rose Grid Registration

• Conditions of Use• Becoming a White Rose Grid user• Obtaining an e-Science certificate

Conditions of Use

• White Rose Grid users must abide by the conditions pertaining to the individual systems at York, Leeds and Sheffield.– http://www.shef.ac.uk/calendar/compregs.html– http://www.leeds.ac.uk/iss/rules/ISShtml– http://www.wrg.york.ac.uk

Becoming a White Rose Grid User1. Register for an account on your local system2. Apply for White Rose Grid resources3. Obtain an e-Science certificate

Why obtain an x509v3 Certificate• Enables secure single sign on to the White

Rose Grid• Use portals e.g. the DAME portal• Use Globus to access WRG compute nodes

Register for an Account on Your Local System

Institution Registration DetailsThe University of Sheffield,Titania

Email: [email protected] Web: http://www.shef.ac.uk/cics/support/userreg/registration.html

The University of Leeds,Maxima,Snowdon

http://www.leeds.ac.uk/iss/helpdesk/ usernames.html

The University of York, Pascali

http://www.wrg.york.ac.uk/access.html

Application for White Rose Grid Resources• Details at http://www.shef.ac.uk/wrgrid/access• Complete the Application form “Application for a

Username and Resources on the White Rose Grid”

• Forward application to local member of the White Rose Grid executive and obtain Authorisation.

• Await confirmation of registration

Accessing the White Rose Grid

• From a UNIX account node access using ssh or XServer application (e.g. Exceed

• Using Globus with e-Science Certificate• White Rose Grid Portal (requires e-Science

certificate)

Accessing the White Rose Grid

• Resources– titan00.shef.ac.uk (10 Sun V880 Sun Fire Servers)– pascali.york.ac.uk (1 Sun V880 Sun Fire Server)– maxima.leeds.ac.uk (5 Sun V880 Sun Fire Servers)– snowdon.leeds.ac.uk (256 node Intel Beowulf cluster)

• From a UNIX account node access using ssh– ssh –l wrsmyname –X [node name].ac.uk

• Allows X applications from maxima and pascali

References

• The White Rose Grid http://www.wrgrid.org.uk• http://www.shef.ac.uk/wrgrid/trainingresources• http://www.shef.ac.uk/wrgrid/access• Grid Support Centre http://www.grid-support.ac.uk• National e-Science Centre http://umbriel.dcs.gla.ac.uk/NeSC• UK DTI e-Science programme http://www.escience-grid.org• Global grid forum http://www.ggf.org

Registration

• Register with the White Rose Grid• Obtain an e-Science Certificate

Go to the linkhttp://www.shef.ac.uk/wrgrid/access/index.html

Grid Computing References

• The Grid: Computing Without Bounds– Ian Foster, Scientific American, April 2003.

• “The Anatomy of the Grid”– http://www.globus.org/research/papers/anatomy.pdf

• Grid Services – “The Physiology of the Grid”– http://www.gridforum.org/ogsi-wg/drafts/

ogsa_draft2.9_2002-06-22.pdf• Research Agenda for the Semantic Grid

– http://www.semanticgrid.org/v1.9/semgrid.pdf

Demonstrators To Try

• http://setiathome.ssl.berkeley.edu/• http://virtualsky.org• http://edg-wp10.healthgrid.org• http://iri02.leeds.ac.uk:9080/damexto/damexto• http://www.opengis.org/resources/?page=demos• http://eepc269.eng.ohio-state.edu/matlab/