The Java CoG Kit - Argonne National Laboratory

/cog

The Globus Alliance contains the following members in alphabetical order:

The Java CoG Kit

Gregor von LaszewskiArgonne National Laboratory

University of Chicago

[email protected]

http://www.cogkit.orgUpdated slides will be available on the CoG Kit web site

/cog

/cog

Funding sources & Acknowledgement

The Java CoG Kit receives funding from thefollowing sponsors DOE MICS NSF NMI

Previous versions of the CoG Kit also receivedfunding from NCSA Alliance

Please, contact [email protected] in case youlike to work with us more closely.

Acknowledgement: CoG Team, Globus Team, Globus Alliance, many

others as listed on www.cogkit.org

/cog

Community

Call on the community to help us withextending and improving the CoG Kit

/cog

Outline

What is the CoG Kit? Basic definitions History of the CoG Kit CoG Kit in action Relationship to GT versions

Selected Project Components Design: Abstractions Programming with Abstractions (Task Graphs) Visual components: Portals & Applications

Conclusion

/cog

Introduction

/cog

Observation

Problem Many application developers desire to program the Grid in

familiar higher level frameworks that allow rapidprototyping.

Solution We propose to reuse a variety of commodity tools,

protocols, approaches, methodologies, while integratingGrid software based on the Globus Toolkit Easier development of advanced Grid services Easier and more rapid application development Easier deployment of Grid services Code reuse and use of component repositories Use of Web services as part of the Grids Widespread use of the Grid Use of commodity technology is not limited to the client!

/cog

Abstractions

Hypothesis: With rapidly changing technologies it may

be beneficial to have an abstraction thatcan be assisting in this technical challenge.

Solution: CoG Kit abstractions are defined for

precisely that reason.

/cog

Result: CoG Kits

CoG Kits make Grid programming simpleand new technologies are easy to integrate

We focus on a CoG Kit for Java Others are possible Python, …

Availability: Java CoG Kit since 1997

Our designs are based on experiencereaching back to the beginnings ofMeta-computing and Grid-computing

/cog

Relationship towards GT

Since GT3 CoG Kit is an essential part of GT

CoG Kit protects from an evolving standard

CoG Kit provides simple programming model

CoG Kit supports portal and GUI developers

CoG Kit is a bridge between application and Gridmiddleware developers.

CoG Kit has known to be working with GT1.0 GT2.4, GT3.0.2, GT3.2, GT3.21, SSH

(under dev.) GT3.9.x, GT4, Condor

(community) Unicore

/cog

Relationship to WS-RF Because …

(Quote: Steve Tuecke, at a GGF meeting):“WS-RF is still under development. The OASIS standardsprocess has just begun.”

COG Kit Provides investment protection while standards are

developed. Provides a more sophisticated programming model than

just services Focus on what many high end-users need You can influence the direction of the CoG Kit by

partnering with us Will work with future versions of GT, SSH, Condor

(planed), … We intend to support and integrate with upcoming new

standards

/cog

History

/cog History (cont.)

CoG Kit was selected by IBM to demonstrate Grid computing in Boards ofDirectors meeting

2001

von Laszewski joins ArgonneNov. 1996

CoG Team: LDAP browser wins Novel developers award1998

CoG Team: The experimental personal gatekeeper of the Java CoG Kit wasbeen able to be installed in less than 30 seconds on a PC including Windows,a similar Globus service installed by an experienced administrator required oneto multiple days.

2000

Term Metacomputer is introduced1992

I-Way1995

Globus Team defines OGSI / Java CoG Kit for GT2.x and GT3.0/OGSI based,includes visual components such as the CoG Kit Desktop, GridFTP interface,GRAM interface

2002

Cog Team: The Java CoG Kit experimental Infogram Service architecture wasdefined combining execution and information Services as a single Grid service.

2001

Term Java CoG Kit is introduced to include jglobus and other components ina single toolkit

1999

Term Grid is introduced1998

Globus version 1 / first release of jglobus based on concepts of protocols and services includes a high throughput fault tolerant workflow prototype

1997

von Laszewski: Graphical Meta-computing environment1994

/cog History

CoG Kit receives best research poster award at SC 20042004

CoG team rewrites the workflow component and introduces GridAnt and anew workflow engine called Karajan that contains flow and structuralcontrol (DAGs, conditions, loops). The workflow concept is expandable.Check pointing and minimal features for fault tolerance are available.

Result caching is possible based on method signatures.

2002 and2003

WSRF is defined2003

A class project shows it is possible to define PBS and LSF providers (notdistributed with the CoG Kit)

2004

Major new Java CoG Kit release.

*GT2, GT3, GT4, SSH providers* Workflow* Graphical components* New manual

2005

CoG team introduces the concept of Grid providers making it possible thatthe CoG Kit can in principal submit to GT2, GT3, GT4, or SSH. Communitydemonstrates also UNICORE provider.

2004

/cog

J a v aJ a v a

C o GC o G

K i tK i t

O G S A / O G S IO G S A / O G S I

G l o b u s T o o l k itG l o b u s T o o l k it

V e r s i o n 2V e r s i o n 2

F i l e T r a n s f e rF i l e T r a n s f e r

G r i d F T PG r i d F T P & R FT & R FT

Mi

dd

le

wa

re

Mi

dd

le

wa

re

Po

rt

al

wa

re

Po

rt

al

wa

re

Ap

pl

ic

at

io

ns

Ap

pl

ic

at

io

ns

Gr

id

Gr

id

S

er

vi

ce

s

Se

rv

ic

es

J o b S u b m i s s i onJ o b S u b m i s s i on

S e c u r i t yS e c u r i t y

W o r k fl o wW o r k fl o w

T a s k M a n a g e m entT a s k M a n a g e m ent

I n f o r m a t i o n I n f o r m a t i o n

S e r v i c e sS e r v i c e s

G l o b u s T o o l k itG l o b u s T o o l k it

V e r s i o n 3V e r s i o n 3

N P A C IN P A C I

G r i d L a bG r i d L a b

OGCEOGCE

G A D UG A D U

A c c e s s G r i dA c c e s s G r i d

C h i m e r aC h i m e r a

G R I P /G R I P / U n i c o r eU n i c o r e

P e g a s u sP e g a s u s

C l i m a t eC l i m a t e

H E PH E P

C h e m i s t r yC h e m i s t r y

G e n o m eG e n o m e

P a r a m e t e r S t u d i e sP a r a m e t e r S t u d i e s

A s t r o n o m yA s t r o n o m y

P r o d u c t i o nP r o d u c t i o n

F r a m ework sF r a m ework s

J a v aJ a v a

P y t h o nP y t h o n

C h e fC h e f

N a n oN a n o M a t e r i a l s M a t e r i a l s

C o GC o G

L a u n c h p a dL a u n c h p a d

C a c t u sC a c t u s

G r i d S p h e r eG r i d S p h e r e

X C A TX C A T

P A C IP A C I

D O ED O E C E R NC E R N

G A F 4 JG A F 4 J

N i m r o d / GN i m r o d / G

G r i d A n tG r i d A n t

P e r lP e r l

C o GC o G

K i tK i t

OGCEOGCE

Use of CoG Kits

/cog

Design

/cog

Design

Based on layered model

Flexible

Expandable

Based on Java interfaces

Abstracts protocols

Abstracts services

Provides workflow

/cog

CoG Kit is more than jglobusJava CoG Kit v4

Documentation Source Community

Unicore prov iderjglobus

abstractions(core)

Web Page

Manual

Wiki

Static pages

Gsissh/term

Cert managementgridfaces

Task management

Swing

portlets

Certificate Authority

Matlab

CoG Workflow/gridant

CoGtop /GridDesktop

CoGShell / Grid Shell

Essential part of GT3.02GT3.2, GT3.2.1GT3.9.x, GT4.0

/cog

CoG Abstraction Layer

CoG CoG CoG CoG CoG

CoG Data and Task Management Layer

CoG Gridfaces Layer

CoG CoG

CoG

GridID

E

GT2GT3OGSIclassic

GT4WS-RF Condor Unicore

Applications

SSHOthersAvakiSETI

Nanomaterials

Bio-Informatics

DisasterManagement Portals

CoG Abstraction Layer

CoG CoG CoG CoG CoG

CoG Data and Task Management Layer

CoG Gridfaces Layer

CoG CoG

CoG

GridID

E

DevelopmentSupport

CoG Abstraction Layers

/cog

Selected Project Components

/cog

Focus on Reusable APIs & Components

Abstractions Provide a simple programming model

Workflow Workflow abstraction

Portals Supporting APIs, abstractions and implementations for

portals. jglobus1.2

GSI security in Java GRAM protocal & client gridFTP protocol & client Myproxy client

Not just API’s but also their implementation

/cogFocus on Abstractions andPatterns

Abstraction above Grid Toolkits Task Model

Jobs, information query, file transfer, authentication,others

Gridfaces model Abstract views of GUIs to the Grid in different

implementations (SWING, JSP, Portlets, …)

Data Types Queues, Sets, Brokers, Schedulers. Based on Task

model

/cog

Java CoG Kit abstractions

A programming model based on a task model thatsimplifies elementary Grid patterns such as jobexecution, file transfer, and file operations.

A programming model that includes executionflows in the form of directed acyclic graphs (DAG).

The programming model is decoupeling thedefinition from the implementation, thusproviding independence from current and futureGrid versions.

Only elementary Grid patterns are considered. It makes programming the Grid simple It makes developing Grid portals more easy Focus is selected functionality

/cog

Design

ExecutableObject

Task

TaskGraph

Handlers

Events

Service

/cogDesign

ExecutableObject

Identity Status

TaskTaskGraph

SecurityContext

ServiceContact

Specification

JobSpecification

FileTransferSpecification

FileOperationSpecification

TaskHandler

*

1

1

1

1

1

1

1

TaskGraphHandler

1

1

1

*

Dependency1

1

* 1

Set Queue1

1

1

1

Service* 1

/cog

Programming with Abstractions

/cog

A simple Programming Patternpublic class COG implements StatusListener{ public void create() { … } public void submit () { … } public void statusChanged (StatusEvent e) { … } public static void main (String arg[]){

try { COG cog = new COG(); cog.create(); cog.submit(); catch (Exception e) { logger.error(“Something went wrong:”, e); }}

/cog

Executing a Simple TaskGraphTaskGraph tg = new TaskGraphImpl();

public void create () { // define tasks ….. /* Add the tasks to the TaskGraph */ tg.add(task1); tg.add(task2); tg.add(task3); tg.add(task4); tg.addDependency(task1, task2); tg.addDependency(task1, task3); tg.addDependency(task2, task4); tg.addDependency(task3, task4);}

public void submit() { TaskGraphHandler handler = new TaskGraphHandlerImpl(); try { handler.submit(tg); } catch (Exception e) { logger.error(``Some Error occured'', e); System.exit(1); }}

Task 1

Task 2

Task 4

Task 3

/cog

Create a taskTask task1 = new Task();

JobSpecification spec = new JobSpecificationImpl();spec.setExecutable(“/bin/ls”);spec.addArguments(“-la”);spec.setStdOutput(“output.txt”);

task1.setSpecification(spec);

// bind the task (late binding)

/cog

Status Monitoringpublic void statusChanged (StatusEvent event) {

Status status = event.getStatus( );

logger.debug(``Status changed to '' + status.getStatusCode());

if (status.getStatusCode( ) == Status.COMPLETED) { logger.info(``Task Done''); elsif (status.getStatusCode( ) == Status.FAILED) { logger.info(``Task Failed''); System.exit(1); }}

Users can design their ownEvent handeling logic basedon status changes

/cog

Using the Handler

try { handler.submit (cog); } catch (InvalidSecurityContextException ise) { logger.error(``Security Exception'', ise); System.exit(1); } catch (TaskSubmissionException tse) { logger.error(``TaskSubmission Exception'', tse); System.exit(1); } catch (IllegalSpecException ispe) { logger.error(``Specification Exception'', ispe); System.exit(1); } catch (InvalidServiceContactException isce){ logger.error(``Service Contact Exception'', isce); System.exit(1); }

DetailedinformationCan be retrievedif exceptions areused

/cog

Bind a Task to a ServiceService service = new ServiceImpl(Service.JOB_SUBMISSION);service.setProvider(``GT3_2_1'');

// Set Security Context – e.g. certificates and suchSecurityContext securityContext =

CoreFactory.newSecurityContext(``GT3_2_1'');securityContext.setCredentials(null); // e.g. set it to default in ./globusservice.setSecurityContext(securityContext);

// Set Contact – e.g. where to go toServiceContact serviceContact = new ServiceContactImpl(

“http://127.0.0.1:8080/ogsa/services/base/gram/ MasterForkManagedJobFactoryService”);service.setServiceContact(serviceContact);

task.setService(Service.JOB_SUBMISSION_SERVICE, service);

ServiceContact serviceContact = new ServiceContactImpl( “http://127.0.0.1:8080”);

/cog

Bind a Task to a ServiceService service = new ServiceImpl(Service.JOB_SUBMISSION);service.setProvider(``GT3_2_1'');

// Set Security Context – e.g. certificates and suchSecurityContext securityContext =

CoreFactory.newSecurityContext(``GT3_2_1'');securityContext.setCredentials(null); // e.g. set it to default in ./globusservice.setSecurityContext(securityContext);

// Set Contact – e.g. where to go to

service.setServiceContact(serviceContact);

task.setService(Service.JOB_SUBMISSION_SERVICE, service);

/cog CoG Kit Desktop

GridShell

JobIcons

MachineIcons

GridLog

NativeIcons

FileTransferGUI

/cog Portlets: OGCE.org

/cog

Contributing

You can contribute

We have a module concept allowingcomponents to be integrated in thedistribution easily

/cog

Conclusion

Programming with CoG abstractions is simple

We envision multiple programming models in CoG

We envision multiple backend services

We can support multiple protocols

We like to engage the community

Contributions: CA management, Unicore provider, gsissh

These contributions are being integrated.