COMP3019 Coursework: Introduction to GridSAM Steve Crouch [email protected], stc@ecs School of Electronics and Computer Science
Dec 31, 2015
COMP3019 Coursework:Introduction to GridSAM
Steve Crouch
[email protected], stc@ecs
School of Electronics and Computer Science
A Reminder…
For materials (inc. slides, software, materials, handout):– http://www.ecs.soton.ac.uk/~stc/
COMP3019/
Coursework deadline: March 27, 2014 at 4pm
GridSAM
Job Submission and Monitoring
GridSAM Overview
What is GridSAM?– A Job Submission and Monitoring Web
Service– Allows compute resources to be
accessible securely over the internet– Uses Job Submission Description
Language (JSDL) to describe requirements for jobs
GridSAM Overview
What is GridSAM to the resource owners?– A Web Service to expose heterogeneous
execution resources uniformly Single machine through Forking or SSH Condor Portable Batch Scheduler (PBS) Load Sharing Facility (LSF) Sun Grid Engine Globus
– Acts as a client to these resources
GridSAM Overview
What is GridSAM to end-users?– A set of end-user tools and client-side APIs
to interact with GridSAM Web Services Submit and start jobs Monitor jobs Terminate jobs File transfer Client-side submission scripting Client-side Java API
GridSAM Server and Client Server comprised of:
– Web Services Container Tomcat/Axis - Web Services provider/container Optional WS-Security - handles authentication
(contains X509 certificate for security) Can be run over HTTP or HTTPS
– GridSAM Java web service
Client comprised of:– Client security ‘container’.
Contains Configuration for security (including X509 certificate security credentials)
Used by client applications to interact with services on OMII server
– GridSAM Java client
Web Service Definition
“A service is the logical manifestation of some physical or logical resources (databases, programs, devices, humans, etc) and/or some application logic that is exposed to the network”
Service interaction is facilitated by message exchanges
Web Services (WS) I
XML: Platform neutral mechanism to describe data
SOAP: Mechanism to describe message exchange
– Simple Object Access Protocol Not simple and nothing to do with Objects!
– Service Oriented Access Protocol Re-engineering of acronym to fit current use!
WSDL: Defines the service interface– Web Services Definition Language
Web Services (WS) II
Services have to reside in a supporting environment:– Called: hosting environment or container
e.g. Axis on Tomcat
– Marshals requests into and response out of the service– Service can discover local configuration parameters– Provides a standard infrastructure for service developers
Processing incoming requests & outgoing responses– Called: Message handlers– Manipulates elements of the message header
Primarily the SOAP header
– Handlers can be applied to message traffic into or out of the whole container or a specific service
Architecture
WS-Security
AXIS
TOMCAT
GridS
AM
…
Client
…
…
GridSAM – Publications & Enabled Activities
GridSAM Architecture A staged event-driven architecture
– Submission pipeline is constructed as a network of stages connected by event queues
– Each stage performs a specific action upon incoming events
Example Pipeline: Condor
Scenario: Condor Pool
GSIFTPGSIFTPFTPFTP WEBDAVWEBDAV HTTPHTTP…
Example using File Staging Objectives: submit simple job with
data input and output requirements and monitor progress
OMII GridSAM
Client
OMII GridSAM
Server
submit JSDL
monitor
OMII GridSAM
FTP Server1 output file
2 input files
JSDL Example <omii_client_home>/gridsam/data/examples/remotecat-staging.jsdl:
<JobDescription>
<JobIdentification> … </JobIdentification>
<Application>
<POSIXApplication xmlns="http://schemas.ggf.org/jsdl/2005/06/jsdl-posix">
<Executable>bin/concat</Executable>
<Argument>dir2/subdir1/file2.txt</Argument>
<Output>stdout.txt</Output>
<Error>stderr.txt</Error>
<Environment name="FIRST_INPUT">dir1/file1.txt</Environment>
</POSIXApplication>
</Application>
…
JSDL Example<DataStaging>
<FileName>bin/concat</FileName>
<CreationFlag>overwrite</CreationFlag>
<Source>
<URI>ftp://ftp.do:55521/concat.sh</URI>
</Source>
</DataStaging>
<DataStaging>
<FileName>dir1/file1.txt</FileName>
<CreationFlag>overwrite</CreationFlag>
<Source>
<URI>ftp://ftp.do:55521/input1.txt</URI>
</Source>
</DataStaging>
<DataStaging>
<FileName>dir2/subdir1/file2.txt</FileName>
<CreationFlag>overwrite</CreationFlag> <Source> <URI>ftp://ftp.do:55521/input2.txt</URI> </Source></DataStaging><DataStaging> <FileName>stdout.txt</FileName> <CreationFlag>overwrite</CreationFlag> <DeleteOnTermination>true</
DeleteOnTermination> <Target> <URI>ftp://ftp.do:55521/stdout.txt</URI> </Target></DataStaging></JobDescription></JobDefinition>