Connecting Workflow-Oriented Science Gateways to Multi-Cloud Systems Zoltán Farkas, Péter Kacsuk, Ákos Hajnal MTA SZTAKI.

Connecting Workflow-Oriented Science Gateways to Multi-Cloud

Systems

Zoltán Farkas, Péter Kacsuk, Ákos HajnalMTA SZTAKI

2

Aim of the paper• Cloud technology has been there now for a while• Continuously deprecating grid middleware technologies• Science gateways enabling easy processing of scientific

computation should be able to make use of cloud technologies as well

• This transition should be completely transparent for the scientists

IWSG’15, 7th International Workshop on Science Gateways

3

Concept of cloud-enabling workflow-oriented science gateways

• Three areas: job submission, data management and security• Abstract architecture for discussing integration possibilities:


Workflow Design

UIWorkfllowStorage

Workflow Enactor

Job Submission ComponentPluginPlugin

4

Job submission possibilities• Necessary actions to enable submission to a given cloud

middleware:o Workflow Design UI extension: new UI components should be created that enable the

cloud-specific configuration of the workflow node (ideally automatically generated based on a template)

o Workflow Storage extension: the Storage should store the cloud-specific properties of workflow nodes as well (ideally automatically serialized)

o Job Submission Component extension: this component should have a new plugin implemented which is able to arrange job submission to the target middleware based on the node’s description in the Storage

• All this performed as many times as many cloud services implementing different services we’d like to support


5

Job submission to multiple cloud systems – Direct method


Workflow Design

UIWorkfllowStorage

Workflow Enactor

Job Submission ComponentPlugin2Plugin1

UI1

UI2

Cloud1 Cloud2

Data1

Data2

Cloud1’

API1 API1 API2

6

Job submission to multiple cloud systems – Intermediate component

• Instead of interfacing with multiple cloud APIs, interface with a single Generic Cloud Access Service (GCAS) which hides the access details of different cloud technologies

• One single extension of the Workflow Design UI is necessary• Single extension of Workflow Storage• One additional plugin in the Job Submission Component


7

Job submission to multiple cloud systems – Intermediate component


Workflow Design

UIWorkfllowStorage

Workflow Enactor

Job Submission ComponentGCAS Plugin

GCAS UI

Cloud1 Cloud2

GCAS

Cloud1’

API1 API1 API2

GCASPlugin1 Plugin2

8

Data handling• Data sooner or later needs to be transferred between the

science gateway and the cloud infrastructure• Direct job submission:

o The data is transferred directly between the cloud storage and the science gateway

• GCAS-based job submission:o Option 1: data is transferred through GCAS (e.g. if no plain HTTP-based access is

available)o Option 2: data transfer is initiated through GCAS, but actual transfer happens between

the cloud storage and the science gateway (e.g. after redirecting HTTP GET requests from GCAS to the cloud storage)


9

Data handling overview


Workflow Design

UIWorkfllowStorage

Workflow Enactor

Job Submission ComponentGCAS Plugin

Cloud1 Cloud2Cloud1’

API1 API1 API2

GCASPlugin1

Plugin2

Opt1Opt2

10

Security considerations• Direct access to cloud services:

o The credentials need to be defined at the science gateway by the usero OR hidden, with the help of robot credentials

• GCAS-based access:o Option 1: GCAS can hide the access credentials of the cloud infrastructures it supports,

thus users need to authenticate only against the GCAS, so one single auth token needs to be provided by the user

o Option 2: GCAS can accept credentials from users, and forward them to the target cloud, so the user needs to define as many credentials as many cloud infrastructures she/he wants to access

o OR, robot credentials can be used to hide GCAS/cloud access credentials


12

Gateway and cloud access use-cases• In an academic environment, the gateway and/or the cloud

infrastructure being used might not be publicly available• In such case applying a single GCAS service might be

problematic


Portal Access Cloud Acccess GCAS used Solution

Public Public Yes or No No special setup needed

Public (or Private, but external)

Private Yes or No Open firewall, use proxy

Private Private Yes Open firewall, use proxy

Private Private No No special setup needed

13

Public gateway, public cloud access


Workflow Design

UIWorkfllowStorage

Workflow Enactor

Job Submission Component GCAS Plugin

Cloud1

API1

GCAS

Plugin1

Plugin1

Services publicly accessibleNo special setup needed

14

Public (or private external) gateway, private cloud access


Workflow Design

UIWorkfllowStorage

Workflow Enactor


Cloud1

API1

GCAS

Plugin1

Pro

xy

Plugin1

Cloud API on a private IPProxy or firewall setup needed

15

Private gateway, private cloud access


Workflow Design

UIWorkfllowStorage

Workflow Enactor


Cloud1

API1

GCAS

Plugin1

Pro

xy

Plugin1

Portal and Cloud API in thesame networkDirect access: no setup neededGCAS: proxy/firewall needed

16

Cloud access implementetion based on GCAS

• Science gateway:WS-PGRADE/gUSE

• GCAS: CloudBroker Platform


17

WS-PGRADE/gUSE Overview• Based on Liferay• General purpose• Workflow-oriented gateway framework• Supports the development and execution of

workflow-based applications• Supports the fast development of domain-

specific gateways by customization methodologies

• Most important design aspect is flexibility


18

WS-PGRADE/gUSE Overview


• DCI Bridge:o Job submission service of WS-PGRADE/gUSEo Accepting job submission requests from the workflow interpreter through a

standardized interfaceo Offering access to different types of middlewares

• Data Avenue:o Data management componento Offers convenient interfaces (web based and API) for managing data located on different

types of storageso Also used by DCI Bridge and WS-PGRADE to provide data for workflow nodes

The CloudBroker Platform• Easy, scalable, secure, integrable and pay-per-use access to scientific and

technical applications in the cloud• High performance computing application store and marketplace with accounting

and billing• Deployment and execution of applications in the cloud with the same

parameters and files as for local execution• Accessible through the internet via a web browser UI• Integration possibilities into third party tools via web service-based APIs• Public version available under https://platform.cloudbroker.com

• Using infrastructure as a service (IaaS) from cloud providers• Offering platform as a service (PaaS) for software vendors• Providing software as a service (SaaS) to end users• Available compute resources: Amazon, IBM, OpenStack, OpenNebula,

CloudSigma• Available storage resources: Amazon, Walrus, Rados S3

https://platform.cloudbroker.com/

CloudBroker Platform Architecture

CloudBroker Platform

AmazonCloud

IBMCloud

…Cloud

ChemistryAppli-

cations

BiologyAppli-

cations

MedicineAppli-

cations

WebBrowser

UI

Engineering Appli-

cations

Web Service API

GenericWorkbenches CloudBroker Integration

Domain-Specific Gateways

R&D End Users and Software Vendors

CLI

…Appli-

cations

21

gUSE and CBP integration• DCI Bridge:

o Extended with a CloudBroker plugino Making use of the Java API of CBP

• WS-PGRADE:o Authentication portlet: allows users to enter their CBP credentials for communicating

with the CBP on behalf of the usero Workflow portlet: workflow node configuration extended with CBP-specific interfaces

• Workflow Interpreter:o Extended with generating JSDL for DCI Bridge based on CBP-specific node properties


WS-PGRADE

gUSE

CloudBroker Platform

Seismology

Proteomics

Rendering

Simulation

Astrophysics

Workflow Portlet

Authentication Portlet

PortalCache

WFIDCI

Bridge

Repository

Rest API Java APIWeb

InterfaceGromac

sBlende

r Blast AutoDock

OpenStackAdapter

OpenNebulaAdapter

Amazon EC2Adapter

Amazon S3Adapter

Rados S3Adapter

OpenStack Cloud

OpenNebula Cloud

Amazon EC2 Rados S3

Amazon S3

Integration features I.• Support for accessing: Amazon EC2, OpenStack, OpenNebula,

CloudSigma• Data transfer happens between WS-PGRADE/gUSE and the

cloud storage – no need to flow data through the CloudBroker Platform

• PaaS-like mode: users can run their own application in the cloud inside a workflow node

• SaaS-like mode: users can select from a set of predeployed applications to be run in the cloud inside a workflow node

Integration features II.• Slight modification in the fully featured UI• Also available in every customization level, completely hiding

the fact that cloud is being used:o ASMo Remote APIo End-user viewo DCI Bridge

WS-PGRADEWF

DeveloperUI

gUSE DCI Bridge

DCI 1

DCI 2

DCI n

ApplicationSpecific

User Interface

ExistingApplicationSpecific UI

WS-PGRADEEnd-User

UI

Remote API

BES interface

ASM API

A

B

C

D

E

BES interface

Integration features III.• Support for commercial clouds with costs (prices configured in

CloudBroker Platform):o Estimated job cost before submissiono Actual job and workflow cost after execution

26

Projects making use of WS-PGRADE/gUSE and CBP integration

• SCI-BUS

• CloudSME


Summary• Presented different methodologies of connecting science

gateways to cloud infrastructures• Focusing on an implementation connecting the

WS-PGRADE/gUSE science gateway framework to cloud infrastructures with the help of the CloudBroker Platform

• The presented implementation enables easy migration of already existing gateways onto the cloud platform – only workflow reconfiguration is necessary

28

Thank you for your attention!

Questions?

The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreements no 283481 (SCI- BUS) and no 608886 (CloudSME).


Connecting Workflow-Oriented Science Gateways to Multi-Cloud Systems Zoltán Farkas, Péter Kacsuk, Ákos Hajnal MTA SZTAKI.

Documents