P-GRADE and WS-PGRADE portals supporting desktop grids and clouds Peter Kacsuk MTA SZTAKI [email protected]
Dec 18, 2015
P-GRADE and WS-PGRADE portals supporting desktop grids and clouds
Peter KacsukMTA [email protected]
2
Parameter study workflow Parameter study workflow execution support in P-GRADE portalexecution support in P-GRADE portal
GEN
SEQ
COLL
SEQSEQSEQ
Parameter sweep jobs:
To be executed in
Grids, Desktop Grids
and Clouds
Generates input
parameter space
Evaluates the results of the
simulation
P-GRADE Portal
3
Motivation: Local DG in Univ. of WestminsterMotivation: Local DG in Univ. of Westminsterbased on BOINC (SZTAKI Desktop Grid)based on BOINC (SZTAKI Desktop Grid)
3
1
2
34
5
6
1. New Cavendish Street 576 nodes2. Marylebone Campus 559 nodes3. Regent Street 395 nodes4. Wells Street 31 nodes5. Little Tichfield Street 66 nodes6. Harrow Campus 254 nodesTotal:
1881 nodes
Lifecycle of a node:1. PCs basically used by students/staff2. If unused, switch to Desktop Grid
mode3. No more work from DG server ->
shutdown (green solution)
4
Generating DG applications
• To port an application to a BOINC system requires three steps:1. Registering the application at the BOINC server2. Creating the master code running on the server3. Creating the client code
• In BOINC Step 2 and 3 require the modification of the original application and this is not trivial
• Using the gUSE/WS-PGRADE environment:– Step 1 by the DG system administrator– Step 2 and 3 are done nearly automatically without any
code modification
5
Automatic generation of master and client code
• To facilitate Step 2 and 3 SZTAKI developed the DC-API (Distributed Computing API) as part of the SZTAKI Desktop Grid (SZDG) package
• DC-API can automatically generate WUs for PS jobs arriving from the workflow
• However, DC-API still requires the modification of the original application to create the client code
• SZTAKI has also developed Genwrapper a generic wrapper that can– eliminate the boincification of the code– automatically generates the client code without the
modification of the original code
6
gUSE architecture and usage
WS-PGRADEWS-PGRADE
Workflow EngineWorkflow Engine
Workflow storageWorkflow storage File storageFile storage
EGEESubmitter
EGEESubmitter
Dedicatedsite
Dedicatedsite
gUSEServices
Meta-brokerMeta-broker
Desktop GridSubmitter
Desktop GridSubmitter
LocalSubmitter
LocalSubmitter
Web ServiceClient
Web ServiceClient
DatabaseClient
DatabaseClient
WMSWMS DG serverDG server WebService
WebService DBMSDBMS
User action, external event or time triggering
7
Connecting gUSE with DGs in CancerGrid
DG ServerBOINCServer
Components
BOINCTaskDB
Sche-duler
Dataserver
QueueManager
DC
-AP
I m
aste
rWU
Job Database(Description of Jobs:Apps, Args, I/O files)
Jobdescr.
gUSE(Workflowenactor)
gUSE DesktopGridgUSE-DG integration
Schedulingpolicy
Batchcreation
gUSEStorage
gUSEWS
Submitter
WS-PGRADE
(User IF)
(WF repre-sentation)
BOINC client
GenWrapper forbatch execution
DC-API cli
LegacyApplication
BOINC client
GenWrapper forbatch execution
DC-API cli
LegacyApplication
gUSE 3G
Bridge SubmitterJob
descr.
gUSELocal
Submitter
8
UoW Local DG
Protein Molecule Simulation using AutoDock
WS-PGRADE portal
9
PS job execution in Grids by P-GRADE
P-GRADE Portal
•3G Bridge
Target Grid
Plugins
•gLite Grid
•ARC Grid
•BOINC Grid
EC2, Eucalyptus
EC2, Eucalyptus
10
P-GRADE portal → 3G Bridge
P-GRADE PortalP-GRADE Portal
FilesFiles
WSClientWSClient
TomcatTomcat
RuntimeRuntime3G
BridgeJobDB
3GBridge
JobDB
Qu
eu
e M
an
ag
er
Grid
Han
dler
Inte
rfac
e
GridPlugin
2
BOINCPlugin
3
CloudPlugin
n
WS
Sub
mitt
er
WS
Sub
mitt
er
GridPlugin
1
DownloadManager
DownloadManager
HT
TP
DHT
TP
D
Submit jobCheck status
GetOutput
11
Host A
3G-Bridge
DC-API – Condor Plugin
Cloud Plugin
Queue 1
……
Queue 2
……
Condor Submitter/ Master
Cloud Resource 2
(Condor Worker)
Cloud Resource N
(Condor Worker)
…
Scheduler
P-Grade Portal
Amazon/ Eucalyptus Cloud Interface
Cloud Resource 1
(Condor Worker) Legend
InformationCommandJob
1. Job is submitted from P-Grade Portal to the 3G-Bridge.
2. 3G-Bridge submits the job to a Condor Cluster using the DC-API Condor Plugin (Queue 1).
• The cluster consists of workers running in the cloud.
3. The Scheduler keeps track of the number of jobs in the Condor cluster and of the number of the running Cloud Resources (workers).
4. If the cluster is overcommitted, the Scheduler starts new workers by submitting a job to the 3G-Bridge queue of the Cloud Plugin (Queue 2).
5. If the cluster under utilized, the Scheduler stops some workers (cloud resources) by cancelling some jobs in the Cloud Plugin Queue (Queue 2).
P-GRADE portal supporting Clouds
12
Conclusions
• By the P-GRADE portal you can submit PS jobs to– Grids– Local clusters– BOINC desktop grids– Clouds
• By writing new target DG plugins you can easily add new type of DG resources
• By writing new target Cloud plugins you can easily add new type of Cloud resources
Thank you