Top Banner
GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute, Germany
29

GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Dec 30, 2015

Download

Documents

Willa Ramsey
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

GridLab: Grid Application Toolkit and

TestbedGabrielle Allen, Ed Seidel and GridLab Team

Center for Computation & Technology, LSU

Albert Einstein Institute, Germany

Page 2: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Ed Seidel, Director

Core funding from state of $9M

Drive new efforts in research, education and economic development

Facilities: supermike, NLR, LONI

Center for Computation & Technology (CCT), LSU

Research groups in CS and computational sciences, e.g.

Grid computing

Computational frameworks

Scientific visualization

Collaborative environments

Page 3: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

GridLab Project

EU Funded by 5th Framework (January 2002):PSNC, AEI, ZIB, MASARYK, VU, SZTAKI, ISUFI, Cardiff, NTUA, Chicago, ISI, Wisconsin, Sun, Compaq

12 Work Packages covering:Grid PortalsMobile UsersDifferent Grid ServicesApplications

Numerical Relativity (Cactus)Gravitational Waves (Triana)

Europe-wide Test Bed

Grid Application Toolkit (GAT)

Page 4: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Grid Application Toolkit (GAT)

Developed through EU GridLab projectApplication oriented access to Grid capabilities through standard API

GATFile_Move(from, to, [details])GATResource_FindResource([details])GAT_LogicalFile(file, name, [details])

Independent of Grid infrastructure and available services.C, C++, Java, [Python], [Perl], [Fortran]API driver for GGF SAGA-RG

Page 5: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

GAT Motivation

Grids and Grid middleware are everywhereGrid applications are lagging behind, big jump from prototypes and demonstrations to real production use of Grids.Problems:

Missing or immature grid servicesChanging environmentDifferent and evolving interfaces to the “grid”Interfaces are not simple for scientific application developers

Application developers accept Grid computing paradigm only slowly

Page 6: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Copy a File: GASS

int RemoteFile::GetFile (char const* source, if (source_url.scheme_type == GLOBUS_URL_SCHEME_GSIFTP || char const* target) { source_url.scheme_type == GLOBUS_URL_SCHEME_FTP ) { globus_url_t source_url; globus_ftp_client_operationattr_init (&source_ftp_attr); globus_io_handle_t dest_io_handle; globus_gass_copy_attr_set_ftp (&source_gass_copy_attr, globus_ftp_client_operationattr_t source_ftp_attr; &source_ftp_attr); globus_result_t result; } globus_gass_transfer_requestattr_t source_gass_attr; else { globus_gass_copy_attr_t source_gass_copy_attr; globus_gass_transfer_requestattr_init (&source_gass_attr, globus_gass_copy_handle_t gass_copy_handle; source_url.scheme); globus_gass_copy_handleattr_t gass_copy_handleattr; globus_gass_copy_attr_set_gass(&source_gass_copy_attr, globus_ftp_client_handleattr_t ftp_handleattr; &source_gass_attr); globus_io_attr_t io_attr; } int output_file = -1; output_file = globus_libc_open ((char*) target, if ( globus_url_parse (source_URL, &source_url) != GLOBUS_SUCCESS ) { O_WRONLY | O_TRUNC | O_CREAT, printf ("can not parse source_URL \"%s\"\n", source_URL); S_IRUSR | S_IWUSR | S_IRGRP | return (-1); S_IWGRP); } if ( output_file == -1 ) { printf ("could not open the file \"%s\"\n", target); if ( source_url.scheme_type != GLOBUS_URL_SCHEME_GSIFTP && return (-1); source_url.scheme_type != GLOBUS_URL_SCHEME_FTP && } source_url.scheme_type != GLOBUS_URL_SCHEME_HTTP && /* convert stdout to be a globus_io_handle */ source_url.scheme_type != GLOBUS_URL_SCHEME_HTTPS ) { if ( globus_io_file_posix_convert (output_file, 0, printf ("can not copy from %s - wrong prot\n", source_URL); &dest_io_handle) return (-1); != GLOBUS_SUCCESS) { } printf ("Error converting the file handle\n"); globus_gass_copy_handleattr_init (&gass_copy_handleattr); return (-1); globus_gass_copy_attr_init (&source_gass_copy_attr); } globus_ftp_client_handleattr_init (&ftp_handleattr); result = globus_gass_copy_register_url_to_handle ( globus_io_fileattr_init (&io_attr); &gass_copy_handle, (char*)source_URL, &source_gass_copy_attr, &dest_io_handle, globus_gass_copy_attr_set_io (&source_gass_copy_attr, &io_attr); my_callback, NULL); &io_attr); if ( result != GLOBUS_SUCCESS ) { globus_gass_copy_handleattr_set_ftp_attr printf ("error: %s\n", globus_object_printable_to_string (&gass_copy_handleattr, (globus_error_get (result))); &ftp_handleattr); return (-1); globus_gass_copy_handle_init (&gass_copy_handle, } &gass_copy_handleattr); globus_url_destroy (&source_url); return (0); }

Page 7: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Copy a File: CoG/RFT

package org.globus.ogsa.gui; TransferRequestType transferRequest = new TransferRequestType (); transferRequest.setTransferArray (transfers1); import java.io.BufferedReader; import java.io.File; int concurrency = Integer.valueOf import java.io.FileReader; ((String)requestData.elementAt(6)).intValue(); import java.net.URL; import java.util.Date; if (concurrency > transfers1.length) import java.util.Vector; { import javax.xml.rpc.Stub; System.out.println ("Concurrency should be less than the number" import org.apache.axis.message.MessageElement; "of transfers in the request"); import org.apache.axis.utils.XMLUtils; System.exit (0); import org.globus.* } import org.gridforum.ogsi.* transferRequest.setConcurrency (concurrency); import org.gridforum.ogsi.holders.TerminationTimeTypeHolder; import org.w3c.dom.Document; TransferRequestElement requestElement = new TransferRequestElement (); import org.w3c.dom.Element; requestElement.setTransferRequest (transferRequest); public class RFTClient { ExtensibilityType extension = new ExtensibilityType (); public static void copy (String source_url, String target_url) { extension = AnyHelper.getExtensibility (requestElement); try { File requestFile = new File (source_url); OGSIServiceGridLocator factoryService = new OGSIServiceGridLocator (); BufferedReader reader = null; Factory factory = factoryService.getFactoryPort (new URL (source_url)); try { GridServiceFactory gridFactory = new GridServiceFactory (factory); reader = new BufferedReader (new FileReader (requestFile)); } catch (java.io.FileNotFoundException fnfe) { } LocatorType locator = gridFactory.createService (extension); Vector requestData = new Vector (); System.out.println ("Created an instance of Multi-RFT"); requestData.add (target_url); TransferType[] transfers1 = new TransferType[transferCount]; MultiFileRFTDefinitionServiceGridLocator loc RFTOptionsType multirftOptions = new RFTOptionsType (); = new MultiFileRFTDefinitionServiceGridLocator(); RFTPortType rftPort = loc.getMultiFileRFTDefinitionPort (locator); multirftOptions.setBinary (Boolean.valueOf ( ((Stub)rftPort)._setProperty (Constants.AUTHORIZATION, (String)requestData.elementAt (0)).booleanValue ()); NoAuthorization.getInstance()); multirftOptions.setBlockSize (Integer.valueOf ( ((Stub)rftPort)._setProperty (GSIConstants.GSI_MODE, (String)requestData.elementAt (1)).intValue ()); GSIConstants.GSI_MODE_FULL_DELEG); multirftOptions.setTcpBufferSize (Integer.valueOf ( ((Stub)rftPort)._setProperty (Constants.GSI_SEC_CONV, (String)requestData.elementAt (2)).intValue ()); Constants.SIGNATURE); multirftOptions.setNotpt (Boolean.valueOf ( ((Stub)rftPort)._setProperty (Constants.GRIM_POLICY_HANDLER, (String)requestData.elementAt (3)).booleanValue ()); new IgnoreProxyPolicyHandler ()); multirftOptions.setParallelStreams (Integer.valueOf ( (String)requestData.elementAt (4)).intValue ()); int requestid = rftPort.start (); multirftOptions.setDcau(Boolean.valueOf( System.out.println ("Request id: " + requestid); (String)requestData.elementAt (5)).booleanValue ()); } int i = 7; catch (Exception e) for (int j = 0; j < transfers1.length; j++) { { System.err.println (MessageUtils.toString (e)); transfers1[j] = new TransferType (); } } transfers1[j].setTransferId (j); } transfers1[j].setSourceUrl ((String)requestData.elementAt (i++)); transfers1[j].setDestinationUrl ((String)requestData.elementAt (i++)); transfers1[j].setRftOptions (multirftOptions); }

Page 8: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Copy a File: GAT/C

#include <GAT.h>

GATResult RemoteFile_GetFile (GATContext context, char const* source_url, char const* target_url) { GATStatus status = 0; GATLocation source = GATLocation_Create (source_url); GATLocation target = GATLocation_Create (target_url); GATFile file = GATFile_Create (context, source, 0); if (source == 0 || target == 0 || file == 0) { return GAT_MEMORYFAILURE; } if ( GATFile_Copy (file, target, GATFileMode_Overwrite) != GAT_SUCCESS ) { GATContext_GetCurrentStatus (context, &status); return GATStatus_GetStatusCode (status); } GATFile_Destroy (&file); GATLocation_Destroy (&target); GATLocation_Destroy (&source); return GATStatus_GetStatusCode (status);}

Page 9: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Copy a File: GAT/C++

#include <GAT++.hpp>

GAT::Result RemoteFile::GetFile (GAT::Context context, std::string source_url, std::string target_url) { try { GAT::File file (context, source_url); file.Copy (target_url); } catch (GAT::Exception const &e) { std::cerr << "Some error: " << e.what() << std::endl; return e.Result(); } return GAT_SUCCESS;}

Page 10: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Copy a File: GAT/C++

#include <GAT++.hpp>

GAT::Result RemoteFile::GetFile (GAT::Context context, std::string source_url, std::string target_url) { try { GAT::File file (context, source_url); file.Copy (target_url); } catch (GAT::Exception const &e) { std::cerr << "Some error: " << e.what() << std::endl; return e.Result(); } return GAT_SUCCESS;}

Page 11: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Copy a File: GAT/C

#include <GAT.h>

GATResult RemoteFile_GetFile (GATContext context, char const* source_url, char const* target_url) { GATStatus status = 0; GATLocation source = GATLocation_Create (source_url); GATLocation target = GATLocation_Create (target_url); GATFile file = GATFile_Create (context, source, 0); if (source == 0 || target == 0 || file == 0) { return GAT_MEMORYFAILURE; } if ( GATFile_Copy (file, target, GATFileMode_Overwrite) != GAT_SUCCESS ) { GATContext_GetCurrentStatus (context, &status); return GATStatus_GetStatusCode (status); } GATFile_Destroy (&file); GATLocation_Destroy (&target); GATLocation_Destroy (&source); return GATStatus_GetStatusCode (status);}

Page 12: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

GAT Solution

GAT API layer between applications and the grid infrastructure:

Higher level than existing grid APIs, hide complexity, abstract grid functionality through application oriented APIsInsulate against

Rapid evolution of grid infrastructureState of Grid deployment

Choose between different grid infrastructuresApplication developers use and develop for the grid independent of the state of deployment of the grid infrastructureService developers can make their software available to many different applications.

Page 13: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Monitoring

Resource Management

InformationSecurity

DataManagement

GLOBUS

ApplicationManager

Logging

NotificationMigration

Profiling

SOAP WSDL Corba OGSA Other

Other GridInfrastructure?

Application

“Copy my file from there to there ..”

Example: Remote File Copying

Page 14: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Example: Remote File Copy

GAT:File.Copy

scpcp gsiscp sftp

gridftpwgetRFT

GridLabProject Specific GridJITnewest

Page 15: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Example: Remote File Copy

GAT:File.Copy

scpcp gsiscp sftp

gridftpwgetRFT

GridLabProject Specific GridJITnewest

#include <GAT++.hpp>

GAT::Result RemoteFile::GetFile (GAT::Context context, std::string source_url, std::string target_url) { try { GAT::File file (context, source_url); file.Copy (target_url); } catch (GAT::Exception const &e) { std::cerr << "Some error: " << e.what() << std::endl; return e.Result(); } return GAT_SUCCESS;}

Page 16: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Grid Application Toolkit

Standard API and Toolkit for developing portable Grid applications independently of the underlying Grid infrastructure and available services

Implements the GAT-APIUsed by applications (different languages)

GAT Adaptors Connect to capabilities/services

Implement well defined CPI (mirrors GAT-API)

Interchangeable adaptors can be loaded/switched at runtime

GAT EngineProvides the function bindings for the GAT-API

Page 17: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

GridLab Architecture

Page 18: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

GAT API Scope

Files

Resources

Events

Information exchange

Utility classes (error handling, security, preferences)

Not more! Keep it simple!

Provide simple functionality which allow us to focus on applications and science.

Page 19: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

GAT API Subsystems

Page 20: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Examples: Read Remote File

try { char data[25];

GAT::File file (context, source_url);

file.open (RD_ONLY); file.seek (100, SEEK_SET); file.read (data, sizeof(data)); file.close ();}catch (GAT::Exception const &e){ std::cerr << "Some error: " << e.what() << std::endl; return e.Result();}

Page 21: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Examples: Read Logical File

try { char data[25];

GAT::LogicalFile logical_file (context, name); list<GAT::File> files = logical_file.get_files ();

files[0].open (RD_ONLY); files[0].seek (100, SEEK_SET); files[0].read (data, sizeof(data)); files[0].close ();}catch (GAT::Exception const &e){ std::cerr << "Some error: " << e.what() << std::endl; return e.Result();}

Page 22: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Example: Spawn a Subtask

GAT::Table sdt; sdt.add ("location", "/bin/date");GAT::Table hdt; hdt.add ("machine.type", "i686");

GAT::SoftwareDescription sd (sdt);GAT::HardwareResourceDescription hrd (hdt);

GAT::JobDescription jd (context, sd, hrd);GAT::ResourceBroker rb (context, prefs);

GAT::Job j = rb.submit (jd);

Page 23: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Implementation

C version fully implemented

C++ version 80% complete

Java version started

Python, Perl, Fortran to follow

Focus: portability, lightness, flexibility, adaptivity

Page 24: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

TFM

TFM TFM TFM TFM

Task Farming on the Grid

TFM implementedin Cactus

GAT used for starting remote TFMs

Designed for the Grid

Tasks can be anything

Page 25: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Black Holes on the Grid

Task farm small Cactus black hole simulations across testbedParameter survey: black hole corotation parameterResults steer a large production black hole simulation

Page 26: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Ubiquitous Computing and Monitoring System (UCoMS)

$2.4M DOE/BOR project

ULL, LSU, Southern, LandmarkPetroleum Engineering, CS, ULL

Deploy sensor networks across Gulf

Data collected to provide input to simulations, tasks farmed out

Results collected (transmitted back)

http://www.ucoms.org

Page 27: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

SURA (NOAA) Project

Data coming in from sensors all over Gulf Realtime Operational Grid

Feeds in to models on SuperMike and other Grid sites

Indication of strengthening storm changing direction: Must Act!

Multiple simulations, expertise, and data connected together: FL, LA, notification, steering sensors

Early warning, preparation:Insurance industry, petroleum industry, local economy all save millions, lives saved

SCOOP: Hurricanes on Grid

QuickTime™ and aPhoto decompressor

are needed to see this picture.

LSU, GoMOOS, Texas A&M,

UaH, UNC, U. Florida,

U. Miami, VIMS

Page 28: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

GridSphere Portal

Portlet based: JSR 168

Collaboration focal point

Interact, share data

Start, monitor remote jobs

Move/browse files

Track and monitor announced jobs

Application specific portlets

e.g.: http://portal.cct.lsu.eduCactus, UCoMS, SCOOP

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Page 29: GridLab: Grid Application Toolkit and Testbed Gabrielle Allen, Ed Seidel and GridLab Team Center for Computation & Technology, LSU Albert Einstein Institute,

Finally

GridLab: http://www.gridlab.org

Grid Application Toolkit

http://www.gridlab.org/GATDocumentation, publications, software download

GridSphere: http://www.gridsphere.org

GGF “Simple API for Grid Applications” (SAGA) Research Group

http://forge.gridforum.org/projects/saga-rg

CCT hiring in all areas: Deputy director, Researchers: Grid computing, computational frameworks, portals, scientific visualization, Cluster and Grid administrators, Application scientists, Graduate students


Related Documents
Seidel STATsites

Seidel STATsites

Category: Documents
Christina Seidel

Christina Seidel

Category: Documents
GridLab Review - WP4 WP4 - Grid Portals GridLab Review Michael Russell russell@aei.mpg.derussell@aei.mpg.de Albert Einstein Institute.

GridLab Review - WP4 WP4 - Grid Portals GridLab Review...

Category: Documents
Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster †, Nicolas Karonis ‡, Matei Ripeanu #, Ed Seidel*, Brian Toonen † * Max-Planck-Institut für Gravitationsphysik.

Gabrielle Allen*, Thomas Dramlitsch*, Ian Foster †,...

Category: Documents
Bernhard Seidel

Bernhard Seidel

Category: Documents
GridLAB-D Technical Support Document: Residential End-Use ...

GridLAB-D Technical Support Document: Residential End-Use...

Category: Documents
GridLab Applications

GridLab Applications

Category: Documents
GridLab, Eger, 31 Mar-1 Apr 2003B.Sathyaprakash@astro.cf.ac.ukp1 Potential Gravitational Applications of Grid B.S. Sathyaprakash GridLab conference, 31.

GridLab, Eger, 31 Mar-1 Apr...

Category: Documents
GridLAB-D - Department of Energyenergy.gov/sites/prod/files/SG 2010 Peer Review - GridL… ·  · 2012-06-04– Model extensions for expected future studies. GridLAB-D ... • Accelerating

GridLAB-D - Department of...

Category: Documents
Samuel W. and Marilyn C. Seidel Seidel School of Education ...Samuel W. and Marilyn C. Seidel Seidel School of Education and Professional Studies 180 ˘˘ ˘ E ˝ ˙ ˘ Pˇ˘ ˆˆ

Samuel W. and Marilyn C. Seidel Seidel School of Education.....

Category: Documents
1 GridLab Grid Application Toolkit and Testbed Contact: Jarek Nabrzyski, GridLab Project Coordinator naber@man.poznan.pl Poznań Supercomputing and Networking.

1 GridLab Grid Application Toolkit and Testbed Contact:...

Category: Documents
Charlette Seidel

Charlette Seidel

Category: Documents