Top Banner
1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas www.csse.monash.edu.au/~davida/ griddles
10

1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas davida/griddles.

Mar 27, 2015

Download

Documents

Cole Wilson
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: 1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas davida/griddles.

1

Flexible IO Services in the Kepler Grid Workflow System

David AbramsonJagan KommineniIlkay Altintas

www.csse.monash.edu.au/~davida/griddles

Page 2: 1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas davida/griddles.

2

KEPLER Core Capabilities Capturing & designing scientific workflows

Making existing (Grid) WFs reusable Composition of services to perform a scientific

WF Executing scientific workflows

Distributed and/or local computation Range of invocation techniques

Local, Web Services, Globus GRAM Various models for computational semantics and

scheduling SDF and PN: Most common for scientific

workflows

Page 3: 1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas davida/griddles.

3

Kepler Directors Orchestrate Workflow Synchronous Data Flow

Consumer actors not started until producer completes

Files copied from producer to consumer. Process Networks

All actors execute concurrently Communication through

TCP/IP Sockets Dedicated IO

IO modes produce different performance results. Actors need to be coded to support specific IO modes

Page 4: 1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas davida/griddles.

4

GriddLeS Legacy applications

Invoked using a range of techniques Need to be shielded from IO details in Grid

Local files Remote files Replicated files Producer-consumer pipes

Don’t want to lock in IO model when application is written (or even Grid Enabled)

Choice of IO model should be Dynamic Late bound

Page 5: 1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas davida/griddles.

5

Flexible IO in GriddLeS

read()write()seek()

open()close()

Local File

Local File

Remote File

Remote File

RemoteApplication

Process

FileMultiplexer

Legacy Application

CacheCache

Late bound decision

ReplicaReplica

SRB

ReplicaReplica

ReplicaReplica

ReplicaReplica

Page 6: 1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas davida/griddles.

6

Integrating Kepler & GriddLeS

Application

Read,

Write,

etc

Grid Buffer Client

Grid Buffer Server

Grid FTP Server

Local File System

Remote File Client

GNS Client

Local File Client

File Multiplexer

SRB Client

GriddLeS NameServer (GNS)

Invoke (Globus, WS, ssh)

Make

Acto

r

Page 7: 1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas davida/griddles.

7

Demo

Page 8: 1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas davida/griddles.

8

Distributed Atmospheric Sciences

CCAM CCAM CCAM

Global Climate DataTemperature, Pressure, etc

Regional Weather DataTemperature, Pressure, etc

All models provided by CSIRO Division of Atmospheric Research

Kepler

Page 9: 1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas davida/griddles.

9

Demo

Page 10: 1 Flexible IO Services in the Kepler Grid Workflow System David Abramson Jagan Kommineni Ilkay Altintas davida/griddles.

10

Acknowledgements CSIRO Division of Atmospheric Sciences

John McGregor, Jack Katzfey and Martin Dix

Funding & Support Australian Research Council (ARC) Australian Government (DCITA, DEST) Hewlett Packard US National Science Foundation US Department of Energy