Fujitsu Laboratories of Europe © 2003 Unicore Technology Dr. David Snelling Grid School July 17, 2003.

Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003

QuickTime™ and aGIF decompressor

are needed to see this picture.

Unicore Unicore TechnologyTechnology

Dr. David SnellingDr. David Snelling

Grid SchoolGrid School

July 17, 2003July 17, 2003




HistoryHistory

HistoryHistoryConceived in 1997UNICORE and UNICORE Plus BMBF ProjectsEuroGrid and GRIP EC ProjectsRealityGrid UK e-Science ProjectSupported Service from Pallas (UnicorePro)

Focus on Seamless ComputingFocus on Seamless ComputingOpen Source and Online Demo SiteOpen Source and Online Demo Site

www.unicore.org

OGSA Compliant DemonstratorOGSA Compliant DemonstratorJune 7th 2002: First release.September 4th 2002: Application Steering Deployment

Fast Track project in RealtyGrid (UK e-Sciecnce).www.sve.man.ac.uk/Research/AtoZ/RealityGrid/AHM2002/

Official OGSI based development under way.




Unicore ImplementationUnicore Implementation

Unicore ArchitectureUnicore ArchitectureOpen Protocol, Abstract Job ObjectReference Implementation (one server, two clients)

Fujitsu Developed ComponentsFujitsu Developed ComponentsSecurity GatewayNetwork Job Superisor (NJS)

Authorization translation, AJO Incarnation, workflow scheduling, file management, ...

Target System Interfaces (TSI)Fujitsu, IBM, NEC, Hitachi, SGI, Sun, Mac, Sony (PS2), ...

Client Tool Kit

Pallas Developed ComponentPallas Developed ComponentProduction Ready GUI Based Client




Important Unicore FunctionsImportant Unicore Functions

SecuritySecurityCertificate/PKI Based Security ModelInteroperable with Globus CertificatesSupport for stronger trust model than Globus

or Globus Trust Model using proxies.

WorkflowWorkflowLocal, remote, and nested task graphsFile management, transfer, and streamingIf and loop constructsFlow control based on task status, time events, and file-state events

GRID ManagementGRID ManagementHard/soft fail recovery.System administrator’s control interface.Extensive logging supportBuilt in certificate management in the GUI client.




Functions ContinuedFunctions Continued

Resource BrokerResource BrokerMulti-site resource check prior to submit.

Estimate of time until execution.

Ticket generation and checking.

Dynamic brokering at execution time.

Interactive Access ExtensionInteractive Access ExtensionAllows standard terminal style interaction.

Unicore Single Sign-on Complete multi-site authentication & authorization.

Includes “Interactive Batch”.




ArchitectureArchitecture

Batch Subsystem

AJO/UPL

User Certificate

Job preparation/control Plugins

Unsafe Internet (SSL)

User authentication

UNICORESite List

UNICOREPro Client

Target System Interface (TSI)

Incarnated job

Commands

User mapping,job incarnation,job scheduling

TSI TSI

Any clustermanagement system

UNICORE SiteFZJ

...

Preparation andControl of jobs

Network Job Supervisor(NJS)

Safe Intranet (TCP)

IDB

Jobs and data transfer to other UNICORE sites

Status request

SV1 Blade files

UUDB IDBIDB

NJS

UNICORE Gateway

optional firewall

optional firewall

AJO/UPL

Runtime Interface

Arcon Client Toolkit User Certificate

UNICORESite List




SGI

Compile

Link

Transfer

Build executable

Distributed JobsDistributed Jobs

Run model

VPPExecute

Export

Get input data

IBMImport

Transfer




AJO: Unicore ProgrammingAJO: Unicore Programming

Abstract Job ObjectAbstract Job ObjectCollection of classes representing Grid functionsEncoded as Java objects (XML encoding possible)

Where to build AJOsWhere to build AJOsPallas client GUI - The user’s viewClient plugins - Grid deployer (You)Arcon client tool kit - Hard core (Some of you)

What can’t the AJO doWhat can’t the AJO doApplication level Meta-computing???




Unicore Can DosUnicore Can Dos

Run & control remoteRun & control remoteShell scriptsApplications (without modifications)

FilesFilesTransfer client <-> serversTransfer servers <-> serversRemote management

Simple VO’sSimple VO’sResourceResource

DiscoveryBrokeringReservation

Provide site autonomyProvide site autonomyData archive accessData archive accessIntegration with local admin.Integration with local admin.

WorkflowWorkflowNested remote jobsNested local jobsLoops &ConditionalsTime based eventsFile state eventsRerun on failure

Seamless ComputingSeamless ComputingFiler streamingFiler streaming

Application to applicationApplication to Client

CORBA IntegrationCORBA IntegrationBuilt in cert. managementBuilt in cert. managementGrid administrationGrid administration

ToolsFailsafe restartDynamic hosting

AJOTour




InterfacesInterfaces

Client PluginsClient PluginsApplication specific support (CPMD, Gausian, NASTRAN, ...)

Unicore Protocol LayerUnicore Protocol LayerResource discovery, job submission, job management

Incarnation Data Base and TSI InterfaceIncarnation Data Base and TSI InterfaceIncarnation support

Unicore User Data BaseUnicore User Data BaseUser mapping and authorization

Broker/Scheduler InterfaceBroker/Scheduler InterfaceInterface defined and implemented in NJS.

Independent File Transfer InterfaceIndependent File Transfer InterfaceInterface defined




The Unicore ClientThe Unicore Client




WorkFlowWorkFlow

Do Construct

If Construct

ComplexDependencies




PluginsPlugins

CPMDPlugin & Wizard




Plugin DevelopmentPlugin Development

Very Seamful: ScriptsVery Seamful: ScriptsDifferent for every system

Seamful: Command TaskSeamful: Command TaskPath to command

Environment variables

Seamless: Appl. ResourceSeamless: Appl. ResourceFully abstracted

No system/site differences

Application metadata

Ideal: PluginIdeal: PluginUser’s view of application

Not a computation,but a

solution

Client DevelopmentClient DevelopmentA few Java classes

GUIPlugin start/stopData container

Use Client FunctionsJob managementResource editor

Examples available

System DevelopmentSystem DevelopmentInstall applicationEdit server data base (IDB)

Application locationEnvironment variablesLibraries and data files




Protocol LayeringProtocol Layering

Abstract Job ObjectAbstract Job Object

Java Zip Stream (File Transfer Only)Java Zip Stream (File Transfer Only)

Unicore Protocol LayerUnicore Protocol Layer

SSLSSL

IPIP

SSL

Unicore Protocol Layer

Java Zip Stream Structure, Integrity, & Compression

Authorization, Packetizing, & Control

Authentication, Integrity, & Encryption

Underlined functions can be turned off.IP

AJO




IDB and TSI InterfacesIDB and TSI Interfaces

Incarnation Data BaseIncarnation Data BaseMaps abstract representation to concrete

Includes resource description

Prototype auto-generation from MDS

Target System InterfaceTarget System InterfacePerl interface to host platform

Very small system specific module for easy porting

Current: NQS (several versions), PBS, Loadleveler, UNICOS, Linux, Solaris, MacOSX, PlayStation-2

Condor: Under development .




UUDBUUDB

Maps user certificate to local loginMaps user certificate to local loginSupport for limited roles.

Support for multiple logins per userSupport for multiple logins per userUser name or project based selection

Simple VO supportSimple VO supportMultiple users to a single account

Selection on user, local account, or projectAll users under a CA

ExtensibleExtensibleExternal interfaceTwo example UUDB implementations available.Can be integrated into existing user management




Resource BrokerResource Broker

Support for simple QoS bidsSupport for simple QoS bidsTurnaround time and price

Ticket basedTicket basedSites retain control of ticket validity.Ticket checking at execution timeAdvanced reservation possible (if site supported)

Planned for 2003Planned for 2003Dynamic site selection supportInteroperable across Unicore and Globus resources




ComputeResourceComputeResource

BrokerBroker

NJSNJSIDBIDB UUDBUUDB

ExpertBrokerExpertBroker

DWDLMExpertDWDLMExpert OtherOther

LocalResourceCheckerLocalResourceChecker

UnicoreRCUnicoreRC GlobusRCGlobusRC

TranslatorTranslator

OntologicalTranslatorOntologicalTranslator

OntologyOntology

SimpleTranslatorSimpleTranslator

MDSGRAM

UoM Broker Architecture

TSI

ICMExpertICMExpert

Look up staticresources

Look upconfiguration

Verify delegatedidentities

Delegate to application-domain expert codeDelegate to Grid architecture-specificengine for local resource check

Pass untranslatable resources to Unicore resource checker

Look up resourcesLook updynamicresources

Delegate resource domain translation

Look up translations appropriateto target Globus resource schema

Broker hosted in NJS

To outside world

Get back set ofresource filters and set ofuntranslatable resources

TicketManagerTicketManager

UNICORE Components

EUROGRID Broker

Globus Components

GRIP Broker

Key:

Inheritance relation

Get signed ticket (contract)

Look up signing identity




Alternate File TransferAlternate File Transfer

Uses independent FTP mechanismsUses independent FTP mechanisms“rcp” or “scp” within an Intranet

GridFTP for sites with relaxed firewall rules

Point to point leased line support

Alternative FTP invisible to usersAlternative FTP invisible to users

Interface extendable to other mechanismsInterface extendable to other mechanisms

FTP Authentication OptionsFTP Authentication OptionsVia Unicore Gateway

Completely independent




AFT ArchitectureAFT Architecture

Vsite A Vsite B

NJS A1

NJS A1

NJS B1

AFTM A AFTM B

UPL

UPL (optional)

ATFM Proprietary

GW B

AB

B




Key TechnologiesKey Technologies

Seamless Computing ModelSeamless Computing ModelJob abstractionIncarnationFile staging, transfer, and streaming support

Security ModelSecurity ModelX509 certificate based security modelConsigner plus endorser modelSeveral CA policies usedFully flexible firewall support

Generic ClientsGeneric ClientsNo changes to application neededClient Tool KitPlugin Interface in GUI Client

Portable ServerPortable ServerJava + Perl




Application Steering ChallengeApplication Steering Challenge

Initial State Gss = 2.0 Gss = 2.0 Gss = 0.5

Gss = 0.5 Gss = 1.8 Gss = 1.8 Gss = 1.8




OGSA RoadmapOGSA Roadmap

UnicoreGateway

Unicore ClientNetwork

JobSupervisor

ResourceDatabase

UserDatabase

Condor

NQS

GT3

ResourceBroker

Multiple firewalllayouts possible

OGSI Style Interfaces (Blue)OGSI Style Interfaces (Blue)Evolve these to standards: “Push and Pull” in the GGF

Key OGSA ServicesKey OGSA ServicesService creation and management

Resource discovery

User authorization

Service orchestration and workflow

Resource brokering

Alternative Client

Ext. AuthService

Fujitsu Laboratories of Europe © 2003 Unicore Technology Dr. David Snelling Grid School July 17, 2003.

Documents

fujitsu laboratories

client slide

unicore technology

clients fujitsu

job incarnation

unicore single sign

gui client

job scheduling tsi