EDGI E uropean D esktop G rid I nitiative

1

2010-04-27

EDGIEDGIEEuropeanuropean DDesktopesktop GGridrid IInitiativenitiativeEDGIEDGIEEuropeanuropean DDesktopesktop GGridrid IInitiativenitiative

gLite job submission to EDGI

EDGI is supported by the FP7 Capacities Programme under contract nr RI-261556

ContentsContentsContentsContents

• Single job submission to EDGI through gLiteSingle job submission to EDGI through gLite

• Metajob submission to EDGI through gLiteMetajob submission to EDGI through gLite

• GBAC submission to EDGI through gLite GBAC submission to EDGI through gLite

For each type of submission, there will be a step-by-step For each type of submission, there will be a step-by-step explanation in order to ease the understanding.explanation in order to ease the understanding.

2

Single job Single job submissionsubmission

3

Step 1: Choose application from the EDGI ARStep 1: Choose application from the EDGI ARStep 1: Choose application from the EDGI ARStep 1: Choose application from the EDGI AR

4

Let us select the dsp app…

Step 2: Get the gridftp url of its glite binary Step 2: Get the gridftp url of its glite binary Step 2: Get the gridftp url of its glite binary Step 2: Get the gridftp url of its glite binary

5

gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dspgsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp

Step 3: Get the VO and site list and selectStep 3: Get the VO and site list and selectStep 3: Get the VO and site list and selectStep 3: Get the VO and site list and select

6

cr1.edgi-grid.eu:8443/cream-pbs-edgidemocr1.edgi-grid.eu:8443/cream-pbs-edgidemo

Step 4: create jdlStep 4: create jdlStep 4: create jdlStep 4: create jdl

• Create your dsp.jdl:Create your dsp.jdl:

Executable = "dsp";Executable = "dsp";

Arguments = "-f 22 -i 22 -p 723 -n pools.txt";Arguments = "-f 22 -i 22 -p 723 -n pools.txt";

InputSandbox = { InputSandbox = {

"gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp" , "gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp" ,

"pools.txt"};"pools.txt"};

OutputSandbox = {"cost.txt"};OutputSandbox = {"cost.txt"};

SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo“;SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo“;

7

Executable reference in the EDGI AR

Input file

Step 5: submit jdlStep 5: submit jdlStep 5: submit jdlStep 5: submit jdl

• Proxy creation:Proxy creation:

> glite-voms-proxy-init -voms > glite-voms-proxy-init -voms edgiprod.vo.edgi-grid.eu:/edgiprod.vo.edgi-grid.eu/Role=edgidemoedgiprod.vo.edgi-grid.eu:/edgiprod.vo.edgi-grid.eu/Role=edgidemo

• Submit:Submit:

> glite-wms-job-submit -a -o id dsp.jdl> glite-wms-job-submit -a -o id dsp.jdl

Connecting to the service Connecting to the service https://wms.grid.edges-grid.eu:7443/glite_wms_wmproxy_server

================== glite-wms-job-submit Success ===================================== glite-wms-job-submit Success ===================

The job has been successfully submitted to the WMProxyThe job has been successfully submitted to the WMProxy

Your job identifier is:Your job identifier is:

https://lb.grid.edges-grid.eu:9000/mhma6si6x__9zyfhG8KCuw

The job identifier has been saved in the following file:The job identifier has been saved in the following file:

/home/lpds/smith/TEST-EDGIDEMO/DSP/id/home/lpds/smith/TEST-EDGIDEMO/DSP/id

================================================================================================================

8

Step 6: get statusStep 6: get statusStep 6: get statusStep 6: get status

• Status:Status:

glite-wms-job-status -i idglite-wms-job-status -i id

================== glite-wms-job-status Success ================== glite-wms-job-status Success ==================================

BOOKKEEPING INFORMATION:BOOKKEEPING INFORMATION:

Status info for the Job : Status info for the Job : https://lb.grid.edges-grid.eu:9000/YFlhyw24Oh6AhZB6Yyk8Gghttps://lb.grid.edges-grid.eu:9000/YFlhyw24Oh6AhZB6Yyk8Gg

Current Status: RunningCurrent Status: Running

Status Reason: unavailableStatus Reason: unavailable

Destination: cr1.edgi-grid.eu:8443/cream-pbs-edgidemoDestination: cr1.edgi-grid.eu:8443/cream-pbs-edgidemo

Submitted: Wed May 16 14:59:48 2012 CESTSubmitted: Wed May 16 14:59:48 2012 CEST

==========================================================================================================

9

Step 7: loggingStep 7: loggingStep 7: loggingStep 7: logging

• Logging:Logging:

glite-wms-job-logging-info -v 2 -i idglite-wms-job-logging-info -v 2 -i id

Check UserTags! It returns:Check UserTags! It returns:

3gbridge_internal_id3gbridge_internal_id

3gbridge_dg_url3gbridge_dg_url

3gbridge_dg_id3gbridge_dg_id

The last one is a URL pointingThe last one is a URL pointingto the WU!to the WU!

10

Step 8: track the job in BOINCStep 8: track the job in BOINCStep 8: track the job in BOINCStep 8: track the job in BOINC

• Inspect BOINC WUInspect BOINC WU

11

Step 9: Step 9: get the get the resultresult

Step 9: Step 9: get the get the resultresult

12

Metajob Metajob submissionsubmission

13

Step 0: prepare your inputsStep 0: prepare your inputsStep 0: prepare your inputsStep 0: prepare your inputs

• Upload your individual input files to web server:Upload your individual input files to web server:

• http://somewhere.com/pools1.txthttp://somewhere.com/pools1.txt

• ……

• http://somewhere.com/pools10000.txthttp://somewhere.com/pools10000.txt

• Create the description of your metajob:Create the description of your metajob:%Required 100%%Required 100%

%SuccessAt 100%%SuccessAt 100%

%Comment pools1.txt%Comment pools1.txt

Arguments = "-i 0 -n pools.txt -f 22 -p 723“Arguments = "-i 0 -n pools.txt -f 22 -p 723“

Input = pools.txt=http://somewhere.com/pools1.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652Input = pools.txt=http://somewhere.com/pools1.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652

QueueQueue

[….][….]

%Comment pools10000.txt%Comment pools10000.txt

Input = pools.txt=http://somewhere.com/pools10000.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652Input = pools.txt=http://somewhere.com/pools10000.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652

QueueQueue

14

Step 1: Choose application from the EDGI ARStep 1: Choose application from the EDGI ARStep 1: Choose application from the EDGI ARStep 1: Choose application from the EDGI AR

15

Let us select the dsp app…

Step 2: Get the gridftp url of its glite binary Step 2: Get the gridftp url of its glite binary Step 2: Get the gridftp url of its glite binary Step 2: Get the gridftp url of its glite binary

16

gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dspgsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp

Step 3: Get the VO and site list and selectStep 3: Get the VO and site list and selectStep 3: Get the VO and site list and selectStep 3: Get the VO and site list and select

17

cr1.edgi-grid.eu:8443/cream-pbs-edgidemocr1.edgi-grid.eu:8443/cream-pbs-edgidemo

Step 4: create and submit jdlStep 4: create and submit jdlStep 4: create and submit jdlStep 4: create and submit jdl

• Create your jdl:Create your jdl:

Executable = "dsp";Executable = "dsp";

Arguments = "-f 22 -i 22 -p 723 -n pools.txt";Arguments = "-f 22 -i 22 -p 723 -n pools.txt";

InputSandbox = { InputSandbox = {

"gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp" , "gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp" ,

"pools.txt","pools.txt",

"_3gb-metajob-dsp-10000" };"_3gb-metajob-dsp-10000" };

OutputSandbox = {"cost.txt"};OutputSandbox = {"cost.txt"};

SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo";SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo";

• Submit:Submit:

glite-wms-job-submit -o idglite-wms-job-submit -o id my-metajob-10000.jdl my-metajob-10000.jdl

18

Reference to the app in the EDGI AR

Basic input file

Extra input file

Step 5: Query the logging and get the urlStep 5: Query the logging and get the urlStep 5: Query the logging and get the urlStep 5: Query the logging and get the url

19

Location of the status description

Step 6: Query the statusStep 6: Query the statusStep 6: Query the statusStep 6: Query the status

20

Metajob handling rules

Actual status of jobs

Location of the mapping

Step 7: Download (and extract) the resultsStep 7: Download (and extract) the resultsStep 7: Download (and extract) the resultsStep 7: Download (and extract) the results

21

•Download result as usual:glite-wms-job-output -i id --dir outputs =>

./outputs/cost.txt

•Extract it:tar zxvf cost.txt =>

./outputs/<subjobid1>/cost.txt

./outputs/<subjobid2>/cost.txt

./outputs/<subjobid10000>/cost.txt

•See the mapping between your individual job definition and the jobids (which gives the name of directories storing the output files of your app) =>

one subjob id

one job

GBAC job GBAC job submissionsubmission

22

EDGIDemoBOINC project

…

OverviewOverview

3G Bridge

…

BOINC Client

GBAC

Application and Inputs

BOINC Client

GBAC

Application and Inputs

Application and Inputs

1. A User submits her application to EDGIDemo trough gLite. Using GBAC as a virtualization framework requires the real binaries and inputs defined as GBAC inputs.

2. Clients (who have VirtualBox installed) will download the BOINC native GBAC application with the submitted application and its inputs.

3. GBAC starts a Linux Virtual Machine (using VirtualBox).

4. GBAC copies the applications and inputs into the Virtual Machine.

5. The application is executed in the Linux VM.6. The result is fetched from the VM by GBAC.7. The VM is shut down and discarded.8. GBAC finishes and the result is returned to

EDGIDemo from the Client.9. The results are returned to gLite through 3G

Bridge from EDGIDemo.10. User can retrieve the results by using gLite

commands

Step 1: Get the gridftp URL of GBACStep 1: Get the gridftp URL of GBACStep 1: Get the gridftp URL of GBACStep 1: Get the gridftp URL of GBAC

• To submit a virtualised app choose the GBAC app from the EDGI ARTo submit a virtualised app choose the GBAC app from the EDGI AR

• Then query the Grid FTP url of the file “gbac.nobin” under the gLite Then query the Grid FTP url of the file “gbac.nobin” under the gLite implementation (as you did it during the single submission)implementation (as you did it during the single submission)

• Gridftp url:Gridftp url:

gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1355/1512/gbac.nobingsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1355/1512/gbac.nobin

24

Step 2: Get the VO and site list and selectStep 2: Get the VO and site list and selectStep 2: Get the VO and site list and selectStep 2: Get the VO and site list and select

25

cr1.edgi-grid.eu:8443/cream-pbs-edgidemocr1.edgi-grid.eu:8443/cream-pbs-edgidemo

Step 3: Create “helloworld” applicationStep 3: Create “helloworld” applicationStep 3: Create “helloworld” applicationStep 3: Create “helloworld” application

• Notes: package must have the name X.app.tgz, where X is the name of Notes: package must have the name X.app.tgz, where X is the name of the executable which must be first launched after unpackaging.the executable which must be first launched after unpackaging.

26

Step 4: create jdlStep 4: create jdlStep 4: create jdlStep 4: create jdl

• Create your jdl:Create your jdl:

Executable = “gbac.nobin";Executable = “gbac.nobin";

Arguments = “anything";Arguments = “anything";

InputSandbox = { InputSandbox = {

"gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1355/1512/gbac.nobin" , "gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1355/1512/gbac.nobin" ,

"myhelloworld.sh.app.tgz" };"myhelloworld.sh.app.tgz" };

OutputSandbox = {"output"};OutputSandbox = {"output"};

SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo";SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo";

27

GBAC reference in the EDGI AR

Your hello world application

Name of GBAC app framework

Step 5: Submit jdlStep 5: Submit jdlStep 5: Submit jdlStep 5: Submit jdl

• Submitting the job:Submitting the job:

glite-wms-job-submit –a –o id helloworld.jdlglite-wms-job-submit –a –o id helloworld.jdl

28

Step 6a: CheckStep 6a: Checkthe status of thethe status of thejob under BOINCjob under BOINC

Step 6a: CheckStep 6a: Checkthe status of thethe status of thejob under BOINCjob under BOINC

29

Step 6b: Waiting for the job to finishStep 6b: Waiting for the job to finishStep 6b: Waiting for the job to finishStep 6b: Waiting for the job to finish

• Get the status of the job:Get the status of the job:

glite-wms-job-status -i idglite-wms-job-status -i id

30

Step 7: get the resultStep 7: get the resultStep 7: get the resultStep 7: get the result

• Get the result:Get the result:

glite-wms-job-output -i id --dir outputsglite-wms-job-output -i id --dir outputs

31

Controlled hands-on has finished at this point, please Controlled hands-on has finished at this point, please play with the tools you learnt or try any of the play with the tools you learnt or try any of the following tasks:following tasks:

Controlled hands-on has finished at this point, please Controlled hands-on has finished at this point, please play with the tools you learnt or try any of the play with the tools you learnt or try any of the following tasks:following tasks:

1.1. Try another application in the EDGI ARTry another application in the EDGI AR

2.2. Assume dsp is an MC type application, submit it 20 times with Assume dsp is an MC type application, submit it 20 times with the same input (“queue 20”)the same input (“queue 20”)

3.3. Take a real application and submit it using GBACTake a real application and submit it using GBAC

4.4. For advanced users: combine MetaJob and GBAC: submit the For advanced users: combine MetaJob and GBAC: submit the helloworld application 10 timeshelloworld application 10 times

a)a) With different inputsWith different inputs

b)b) With the same inputWith the same input

32

Thank you for your attention!Thank you for your attention!

www.edgi-project.euwww.edgi-project.eu

33