Community Software Development with the Astrophysics Simulation Collaboratory Authors: Gregor von Laszewski, Michael Russell, Ian Foster, John Shalf, Presenter:

Community Software Development with the Astrophysics Simulation Collaboratory

Authors: Gregor von Laszewski, Michael Russell, Ian Foster, John Shalf,

Presenter: Javier Munoz Agnostic: Ana Rodriguez

Gabrielle Allen, Greg Daues, Jason Novotny, Edward Seidel

Outline Introduce the ASC

Cactus: Architecture

Cactus: Math

Cactus: Scaling Out

Gridsphere

Agnostic Questions

Outline Introduce the ASC

Cactus: Architecture

Cactus: Math

Cactus: Scaling Out

Gridsphere

Agnostic Questions

NSF Award Abstract -#9979985KDI: An Astrophysics Simulation Collaboratory: Enabling Large Scale Simulations in

Relativistic Astrophysics

NSF Org PHY Latest Amendment Date September 22, 2003 Award Number 9979985 Award Instrument Standard Grant Program Manager Beverly K. BergerPHY DIVISION OF PHYSICSMPS DIRECT FOR MATHEMATICAL & PHYSICAL SCIEN Start Date September 15, 1999 Expires August 31, 2004 (Estimated) Expected

Total Amount $2,200,000.00 (Estimated) Investigator

Wai-Mo Suen [email protected] (Principal Investigator current)Ian Foster (Co-Principal Investigator current)Edward Seidel (Co-Principal Investigator current)Michael L. Norman (Co-Principal Investigator current)Manish Parashar (Co-Principal Investigator current)

Sponsor Washington UniversityONE BROOKINGS DRIVE, CAMPUS BOXSAINT LOUIS, MO 631304899 314/889-5100 NSF Program 8877 KDI-COMPETITION

Astrophysics Simulation Collaboratory (ASC)

Astrophysics General Theory of Relativity

Simulation Numerical solution of Partial

Differential Equations

Collaboratory Infrastructure to support

efforts to solve large complex problems by geographically distributed participants.

Tired of Middleware?

The ASC is a complete Application

BUT we’ll talk middleware anyway…

Focus

Application (70%)

(30%)Middleware

ASC: Purpose Community (VO) Domain-specific

components Transparent access Deployment services Collaboration during

execution Steering of simulations Multimedia streams

ASC: Technologies Used Cactus Framework

Application Server

Grid Tools

Cactus Simulations in the ASCportal (Agnostic 4)

www.ascportal.org

Outline Introduce the ASC

Cactus Architecture (Agnostic 1)

Cactus: Math

Cactus: Scaling Out

Gridsphere

Agnostic Questions

Cactus 1995 Original version:

Paul Walker Joan Masso Edward Seidel John Shalf

1999 Cactus 4.0 Beta 1 Tom Goodale Joan Masso Gabrielle Allen Gerd Lanfermann John Shalf.

Why Cactus?

Parallelization Model Easy to Grid-enable Flexible C and Fortran

Cactus

Modularity New Equations (physics) Efficient PDE solution (Numerical Analyst) Improved distributed algorithm (CS)

Cactus Building Blocks:

Schedule Driver Flesh Thorns Arrangements Toolkit

www.cactuscode.org

Scheduling (workflow) Flesh invokes a

driver to process the schedule

ww.cactuscode.org

Driver Parallelizes the execution Management of grid variables

storage distribution communication

Distributed memory model.  section of the global grid, Boundaries: Physical or Internal

Each thorn is presented with a standard interface, independent of the driver.

Driver PUGH (Parallel Unigrid Grid Hierarchy)

MPI Uniform mesh spacing Non-adaptive

Automatic grid decomposition Manual decomposition

Number of processors in each direction Number of grid points on each processor

Flesh

In general, thorns overload or register their capabilities with the Flesh, agreeing to provide a function with the correct interface

Thorn: Anatomy Cactus Configuration

Language Parameter Files (Input) Configuration File

Interface Schedule

Application Code C Fortran

Miscellaneous Documentation Make

ww.cactuscode.org

Thorn: Files

Param.ccl: What are the parameters for my thorn? What are their ranges? Are they steerable? What parameters do I need from other thorns? Which of my parameters should be available for other thorns?

Interface.ccl: What does my thorn “do” What are my thorns grid variables? What variables do I need from other thorns? What variables am I going to make available to other thorns? Timelevels Ghostzones

Schedule.ccl: When and how should my thorns routines be run? How do my routines fit in with routines from other thorns? Which variables should be synchronized on exit?

Objectives Introduce the ASC

Cactus: Architecture

Cactus: Math

Cactus: Scaling Out

Gridsphere

Agnostic Questions

Finite Differencing: Infinitesimal vs Small Delta

http://homepage.univie.ac.at/franz.vesely/cp_tut/nol2h/applets/HO.htmlhttp://homepage.univie.ac.at/franz.vesely/cp_tut/nol2h/applets/HO.html

Finite Differencing: Ghost Zones

The grid on each processor has an extra layer of grid-points (in blue) which are copies of those on the neighbor in that direction

After the calculation step the processors exchange these ghost zones to synchronize their states.

Cactus 4.0 User’s Guide

Finite Differencing: Time Levels

Similar to Ghost Zones for the time dimension

Cactus managed leads to optimization Numerical differentiation algorithm

dependent Typically three

Finite Differencing: Synchronization

Cost of parallelization Network characteristics important Transfer of 12MBytes per iteration

BUT…there is room for optimization

Objectives Introduce the ASC

Cactus: Architecture

Cactus: Math

Cactus: Scaling Out

Gridsphere

Agnostic Questions

Cactus at Work Members of the Cactus and Globus projects after

winning a Gordon Bell Prizes in high-performance computing for the work described in their paper: Supporting Efficient Execution in Heterogeneous Distributed Computing Environments with Cactus and Globus

What did they do?

Scaled Out Grid enabled four supercomputers 249 GFlops

Efficiently Scaling efficiency:

88% with 1140 CPU’s 63% with 1500 CPU’s

Scaling Out Finite Differencing Solutions to PDE’s

Problem: Nodes with different types of Processors,

Memory sizes Heterogeneous Communications among

processors. Multiprocessors LAN WAN Bandwidth, TCP, and Latency

Scaling Out in a Computational Grid

Strategies Irregular data distribution Grid-aware communication schedules Redundant computation Protocol tuning

Where are they now? (Agnostic 5,8)

Gabrielle AllenAssistant Director for Computing Applications, Center for Computation & Technology

Associate Professor, Department of Computer Science Louisiana State University Edward Seidel

Director, Center for Computation & TechnologyFloating Systems Professor, Departments of Physics and Computer ScienceLouisiana State University

Visualization of Katrina developed at CCTApplication Frameworks for High Performance andGrid Computing, G. Allen, E. Seidel, 2006. http://www.cct.lsu.edu/~gallen/Preprints/CS_Allen06b.pre.pdf

Outline Introduce the ASC

Cactus: Architecture

Cactus: Math

Cactus: Scaling Out

Gridsphere

Agnostic Questions

GridSphere

1st Grid Middleware Congresswww.GridSphere.org

GridSphere (Agnostic 3)

Developed by the EU GridLab project About 100,000 lines of code Version 2.0

Framework based on: Grid Portal Development Kit (GPDK) ASC Web Portal

Open source project http://www.gridsphere.org

Framework for portlet development Portlet Container

Ongoing Collaborations (Agnostic 10)

Cactus portal at Albert Einstein Institute Interface to Cactus numerical relativity application /

provide physicists with interface for launching jobs & viewing results

Grid Portal at Canadian National Research Council Provide controlled remote access to NMR spectroscopy

instruments GEON earth sciences portal / CHRONOS portal

Manage/visualize/analyze vast amount of geosciences data and large scale databases

Pgrade portal at SZTAKI Hungary & U. Westminster UK Creation, execution and monitoring of complex workflows

GridSphere Core Portlets

Login/Logout Role Based Access Control (RBAC) separating users into

guests, users, admins, and super users Account Request Account Management User Management Portlet Subscription Local File Manager Notepad Text Messaging

Action Portlets

Hides branching logic Action and view methods to invoke for

events Provides default actionPerformed

Personalized Environment

"GridSphere’s Grid Portlets A Grid Portal Development Framework" Jason NovotnyGridSphere and Portlets workshop, March 2005, e-Science Institute

Single Sign-On Capabilities

"GridSphere’s Grid Portlets A Grid Portal Development Framework" Jason NovotnyGridSphere and Portlets workshop, March 2005, e-Science Institute

Perform File Transfers (Agnostic 2)

"GridSphere’s Grid Portlets A Grid Portal Development Framework" Jason NovotnyGridSphere and Portlets workshop, March 2005, e-Science Institute

GridSphere

"GridSphere’s Grid Portlets A Grid Portal Development Framework" Jason NovotnyGridSphere and Portlets workshop, March 2005, e-Science Institute

Submit Jobs

"GridSphere’s Grid Portlets A Grid Portal Development Framework" Jason NovotnyGridSphere and Portlets workshop, March 2005, e-Science Institute

GridSphere

https://portal.cct.lsu.edu/gridsphere/gridsphere?cid=home

Outline Introduce the ASC

Cactus: Architecture

Cactus: Math

Cactus: Scaling Out

Gridsphere

Agnostic Questions

Agnostic Questions6. The ASC application server uses a relational database to

maintain the state of sessions; could it be implemented in any other way? Explain.

Sure, SQL is a de-facto standard; The Gridsphere Container provides a Persistence Manager that uses open-source Castor libraries from Exolab which provides mechanisms for mapping objects to SQL and an object query language (OQL)

Using Castor, mappings from Java to SQL can be generated automatically

Agnostic Questions7. Can you expand on the MDS browser developed by the

Java CoG Kit?

Currently MDS4 uses XPATH instead of LDAP for query language. Registration is performed via a Web service Container built-in MDS-Index service Aggregator services are dynamic A Service can become Grid wide service index.

Source: http://globus.org/toolkit/docs/4.0/key/GT4_Primer_0.6.pdf

Agnostic Questions9. Could the collaboratory framework be implemented using

technologies other than Java? If so, could it still be used in the same way? What would be the pros and cons of using Java technologies vs. other alternative technologies?

Cactus uses C and FORTRAN, but Web Portals are mainly being developed using Java:

GridPort GCE-RG GPDK GridSphere

Questions?

Fallacy? “Discipline scientists are typically not

experts in distributed computing.”

Cactus Developers have a background in Mathematics and Physics.