GT4 GRAM: A Functionality and Performance Study Stuart Martin, Martin Feller Computational Institute, University of Chicago & Argonne National Lab TeraGrid.

GT4 GRAM: GT4 GRAM: A Functionality and A Functionality and Performance StudyPerformance Study

Stuart Martin, Martin FellerStuart Martin, Martin FellerComputational Institute, University of Computational Institute, University of

ChicagoChicago & Argonne National Lab & Argonne National Lab

TeraGrid 2007TeraGrid 2007

Madison, WIMadison, WI

2

Contributors / CollaboratorsContributors / Collaborators

UC/ANL– Ian Foster– Peter Lane– Jarek Gawor– Ravi Madurri– Rachana Ananthakrishnan

3

GRAM - Basic Job GRAM - Basic Job Submission and Control ServiceSubmission and Control Service

A uniform service interface for remote job submission and control– Includes file staging and I/O

management– Includes reliability features– Supports basic Grid security

mechanisms– Asynchronous monitoring– Interfaces with local resource

managers, simplifies the job of metaschedulers/brokers

GRAM is not a scheduler.– No scheduling– No metascheduling/brokering

5

ComparisonComparison

Functionality– Security

– File Staging

– General Performance

– Concurrent jobs

– Sequential jobs

6

Security Functional ComparisonsSecurity Functional Comparisons

7

Privilege Limiting ModelPrivilege Limiting Model

GRAM must be able to start jobs submitted by remote users under different user ids. It must execute some code as “root”– GRAM2: Entire gatekeeper runs as root

– GRAM4: Service with sudo privs> non-root container account requires sudo to invoke operations as other users

8

AuthenticationAuthentication

A client can authenticate with GRAM with a variety of protocols– GRAM2: TLS (only)

– GRAM4: TLS, Message Level Security> Message-level WS-Security

> Channel-level WS-SecureConversation

> Choice for which to support in each deployment

9

Credential DelegationCredential Delegation

Needed by GRAM or the user’s applications to do file staging or other grid operations– GRAM2: Yes, Required

> Clients must delegate from client to service on every request

– GRAM4: Yes, Optional> Clients can choose and delegate when necessary

10

Credential RefreshCredential Refresh

Credentials have a lifetime and may expire before a job has completed execution– GRAM2: Yes– GRAM4: Yes

> A client can query for information about the WS Resource of the delegated credential

> Remaining lifetime

11

Share credential delegationShare credential delegationamong jobsamong jobs

When repeatedly interacting with the same GRAM service, a client may want to delegate once and share the delegation among multiple jobs– GRAM2: No– GRAM4: Yes

> Refreshing a credential in the delegation service that was shared among multiple job submission will results in a refresh for each job

12

Authorization CalloutsAuthorization Callouts

Following authentication, GRAM checks to see if the request should be authorized.For example, a gridmap file acting as an access control list– GRAM2: Yes - single PDP callout– GRAM4: Yes - Multiple PDP callout chain> Allows for richer policies

Parse VOMS attributes Use attributes in policy evaluations Site level black lists

13

File ManagementFile Management Functional Comparisons Functional Comparisons

14

File StagingFile Staging

Job staging before and after the user’s job is executed– GRAM2: Yes– GRAM4: Yes

15

File staging retry policyFile staging retry policy

If a file staging operation fails, it may be non-fatal and retry may be desired– GRAM2: None– GRAM4: RFT Supported

> Server defaults for all transfers can be configured

> Defaults can be overridden for a specific transfer

16

Incremental output staging Incremental output staging “streaming”“streaming”

It can be useful to obtain access to data produced by a program as it executes.– GRAM2: stdout/stderr only– GRAM4: stdout/stderr and any file

> A client can stream files via the service-side GridFTP server. This is what globusrun-ws does for stdout and stderr streaming.

17

Standard input accessStandard input access

The contents of a file can be passed to the job’s standard input– GRAM2: Yes– GRAM4: Yes

18

Throttle staging workThrottle staging work

A GRAM submission that specifies file staging imposes load on the service node executing the GRAM service.– GRAM2: No– GRAM4: Yes

> GRAM is configured for a maximum number of “worker” threads and thus a maximum number of concurrent staging operations.

19

Load balance staging workLoad balance staging work

Allow staging work to be load balanced among a set of service hosts– GRAM2: No– GRAM4: Yes

> Staging work can be distributed over several “service nodes”. For example, a separate GridFTP server can be configured for each LRM type or file system paths.

20

GeneralGeneral Functional Comparisons Functional Comparisons

21

Access protocolAccess protocol

Protocol used to interact with the service– GRAM2: proprietary HTTP– GRAM4: Web Service SOAP

> Standards based WSDL Client tooling

22

Job Description LanguageJob Description Language

The mechanism for specifying job directives.– GRAM2: RSL

> Custom string-based language

– GRAM4: JDD> Job description document (JDD) XML-based version

> Initial prototype of OGF’s JSDL specification

23

Extensible Job Description Extensible Job Description LanguageLanguage

A mechanism for passing “extensions” through GRAM to underlying local resource managers– GRAM2: Yes– GRAM4: Yes

24

Local Resource Manager InterfaceLocal Resource Manager Interface

The GRAM interface to the LRM to submit, monitor, and cancel jobs.– GRAM2: Perl scripts– GRAM4: Perl scripts + SEG

> Scheduler Event Generator (SEG) provides efficient monitoring between the GRAM service and the LRM for all jobs for all users

25

Local Resource ManagersLocal Resource Managers

Supports a range of LRMs - PBS, LSF, Condor, Fork, …– GRAM2: Yes– GRAM4: Yes

26

Fault ToleranceFault Tolerance

GRAM can recover from a container or host crash. Upon restart, GRAM will resume processing of the users job submission– GRAM2: Yes - Client initiated

> Processing resumes for a single job after the client has restarted the job manager service process

– GRAM4: Yes - Service initiated> Processing resumes for all jobs once the service container has been restarted

27

State Access: Push State Access: Push (subscription)(subscription)

Allow clients to request notifications for state changes– GRAM2: Yes - callbacks– GRAM4: Yes - WS Notifications

> Clients can subscribe for notifications to the “job status” resource property

28

State Access: PullState Access: Pull

Allow clients to get the state for a previously submitted job– GRAM2: Yes

> The service defines a proprietary operation to get the job state.

– GRAM4: Yes> The service defines a WSRF resource property that contains the value of the job state. A client can then use the standard WSRF getResourceProperty operation.

29

Audit LoggingAudit Logging

Allow an audit records to be inserted into an audit DB when a job completes– GRAM2: Yes– GRAM4: Yes

> An enhancement was contributed by Gerson Galang (APAC) to insert the record at the beginning of the job and to update the audit record after submission and again at job end.

30

At Most Once Job SubmissionAt Most Once Job Submission

A simple request-reply job submission protocol has the problem that if the reply message is lost, a client cannot know whether a job has been started. Measures need to be taken to ensure that the same job is not submitted twice.– GRAM2: Yes - 2-phase commit

> Requires an extra round trip, plus a delay on the service to begin processing

– GRAM4: Yes - UUID on create> The client supplies a client-created unique ID (UUID) and the GRAM4 service guarantees not to start a job with a duplicate ID

31

Job CancellationJob Cancellation

Allow a job to be cancelled– GRAM2: Yes

> Proprietary operation

– GRAM4: Yes> WSRF standard “Destroy” operation

32

Job Lifetime ManagementJob Lifetime Management

Allow a client to control when a job’s state is cleaned up– GRAM2: Yes

> Implements a set of job directives and operations

– GRAM4: Yes> Standard WS-ResourceLifetime operations

33

Maximum Active JobsMaximum Active Jobs

The Maximum number of jobs that the service can manage– GRAM2: ~250

> Due to each job Job Manager process querying the LRM separately

– GRAM4: 32,000> Limited by the number of directories that can be created in a directory

34

Parallel Job SupportParallel Job Support

Support for MPI jobs “jobtype = MPI”– GRAM2: Yes– GRAM4: Yes

35

MPICH-G SupportMPICH-G Support

Support for multi-site MPI– GRAM2: Yes

> Client-side DUROC and service-side DUCT service

– GRAM4: Yes> Multi-job and rendezvous Web Services> MPIg support coming soon

36

Basic Execution Service (BES) Basic Execution Service (BES) InterfaceInterface

Support for OGSA BES for job submission– GRAM2: No– GRAM4: Prototyped

> Working on plans to initially support JSDL with the current GRAM4 port type, then add support for BES too

37

Performance ComparisonsPerformance Comparisons

38

Concurrent JobsConcurrent Jobs(as in paper)(as in paper)

Stage

In

Stage

Out

File Clean Up

Unique Job Dir

GRAM2 GRAM4

None None No No 2552 2100

1X10KB 1X10KB No No 2608 3779

1X10KB 1X10KB Yes Yes 2698 5695

Average seconds per 1000 jobsCondor-g to GRAM to Condor LRM

39

Concurrent JobsConcurrent Jobs(as will be in GT 4.0.5)(as will be in GT 4.0.5)

Stage

In

Stage

Out

File Clean Up

Unique Job Dir

GRAM2 GRAM4

None None No No 2552 2176

1X10KB 1X10KB No No 2608 2147

1X10KB 1X10KB Yes Yes 2698 2254

Average seconds per 1000 jobsCondor-g to GRAM to Condor LRM

40

Improving performance forImproving performance forstaging jobsstaging jobs

Adding local method call mechanism for general use in Java WS Core (4.0.5)– GRAM is doing this with RFT– Any service which calls another in-process service could make similar modifications for local calls and likely benefit from improved performance

Adding caching of the GridFTP server connections in RFT (4.0.6)

41

Sequential JobsSequential Jobs

Delegation

Stage

In

Stage

Out

GRAM2 GRAM4

None None None N/A 1.70

Per Job None None 1.07 3.53

Per Job 1X10KB None 1.78 5.57

Shared 1X10KB None N/A 5.41

Per Job 1X10KB 1X10KB 2.44 9.08

Shared 1X10KB 1X10KB N/A 7.91

Average seconds per job (Fork)

42

Sequential JobsSequential Jobs

Delegation

Stage

In

Stage

Out

GRAM2 GRAM4

None None None N/A 1.46

Per Job None None 1.07 3.42

Per Job 1X10KB None 1.78 3.46

Shared 1X10KB None N/A 3.51

Per Job 1X10KB 1X10KB 2.44 5.25

Shared 1X10KB 1X10KB N/A 3.67

Average seconds per job (Fork)

43

For More InformationFor More Information

Stuart Martin - smartin@mcs.anl.gov Martin Feller - feller@mcs.anl.gov