SZTAKI & International Desktop Grid Federation : status report

11

SZTAKI & International Desktop Grid Federation: status report

József Kovács (MTA SZTAKI)[email protected]

http://idgf-sp.eu

IDGF-SP is supported by the FP7 Capacities Programme under contract nr RI-312297

2

International Desktop Grid Federation: short introduction

Legal form: Dutch foundationMain vehicle for dissemination during the lifetime of the EC funded projects and for continuation after the projects Federation is open for organisations and individuals in research and industry A member organisation with chapters and working groupsFocal point for DCI/Desktop operators and application developersWebportal: discussion fora, events and workshops, support tools, etc.Provide assistance and facilities to launch new joint actions, initiatives, projects

3

Involve and engage in long-term significantly more citizens and new communities in the • volunteer and • private (campus-wide or

enterprise) Distributed Computing Infrastructures

by supporting the (1) rapid creation, (2) efficient operation, and (3) dynamic expansion of this type of DCIs for e-Science. Coordinate and synchronise the dissemination and support activities of major European stakeholders of volunteer and Desktop Grids with focus on the International Desktop Grid Federation.

IDGF-SPInternational Desktop Grid Federation - Support Project

Supported by: EC in the FP7Project acronym:

IDGF-SPContract n°: RI-312297Project type: CSA-SAStart date: 01/11/2012Duration: 24 months

Funding from the EC: 860.000 €

Total funded effort in person-month: 136

Web site: http://idgf-sp.euCoordinator:

Dr. Robert Lovasemail: [email protected]

4

SUMMARY: software components of maintained by SZTAKI

• SZDG boinc package: debian variant of BOINC supported by our tools

• 3G Bridge: remote job submission, converting incoming jobs to WUs

• DC-API: c library for creating master-worker applications• Genwrapper: porting legacy application to BOINC, shell

interpreter• GBAC: virtualisation under BOINC client, based on virtualbox• Metajob: submission of large number of jobs to 3Gbridge in

one turn• Cloud handling by 3G Bridge: dynamic on-demand cloud

bursting• Monitoring by 3G Bridge: information collection on job

throughput• EGI: gLite CREAM mCE: automatic job transfer from gLite to

BOINC• EGI: Munin and Nagios: EGI compliant monitoring• gUSE: web-based graphical portal for executing jobs and

workflows

4

5

Documentation site: doc.desktopgrid.hu

5

6

Recent (ongoing) activities and plans (I.)

• Supporting the latest BOINC release by our tools (DC-API, 3GBridge, …)• plan is to put DC-API into the BOINC trunk• testing has been successfully done• need David to integrate our staff into the BOINC code repo

• Improving GBAC• handling of stdout & stderr of application has been improved• returning the log of the internal GBAC component for debugging purposes• automatic image extraction on the client side (image size 700MB -> 100MB)• short term plan is to support image selection (discussing what is the

easiest/most efficient way to implement it…) • Improving 3GBridge

• 3GBridge supports REST interface, implementation finished• http://doc.desktopgrid.hu/doku.php?id=manual:restif• consumed CPU hours, credits can now be queried (required for EGI

accounting & ELTE)6

7

Recent (ongoing) activities and plans (II.)

• Volunteer Storage• Using random linear coding in distributed storage• See presentation from Adam Visegradi

• New applications on our public, volunteer projects• BIOVEL community applications on EDGeS@home

• Biome-BGC ecosystem modeling is the estimation of behavior of terrestrial ecological systems (forests, grasslands, agricultural areas) as accurately as possible.

• New mathematical application will appear on SZTAKI Desktop Grid these days• Riemann zeta research project: http://www.riemann-siegel.com/• Locating large values of zeta functions on the critical lines

• New user community: DIRAC• The DIRAC system has been recently integrated with BOINC through GBAC• The GBAC VM image has been adjusted to the needs of DIRAC• DIRAC jobs are currently arriving through gLite• Testing finished, deployment on EDGeS@home will happen these days 7

8

Recent (ongoing) activities and plans (III.)

• Cloud scenarios• We created cloud images to support automatic setup of BOINC server with

clients• To ease the deployment• To perform dynamic, on-demand computing infrastructure for scientific users• To encourage scientific users to utilise BOINC

• We are creating images for each scenario defined at doc.desktopgrid.hu• Image with BOINC client

• Needs a proper configuration to force the client to continuously execute WUs in the cloud

• Not so easy in the new scheduling algorithm, recompilation is needed• Adjusting related parameters hardwired in the client_state.h would help…

• #define WORK_FETCH_PERIOD 60• #define WF_MIN_BACKOFF_INTERVAL 600• #define WF_MAX_BACKOFF_INTERVAL 86400• #define WF_DEFER_INTERVAL 300

8

9

European Grid Infrastructure(short overview)

• European• Over 35 countries

• Grid• Secure sharing of IT

resources• Infrastructure

• Computers (clusters): 372,612 CPUs

• Data: 180 PB disk• Applications: 1.67 million

job/day (thousands of scientists)• …. and beyond!!

• Technologies to • Federate resources• Access resources 9

Coordinator: EGI.eu,

Amsterdam

10

Proposed EGI DG-VT Project

• General project info• Start (planned) : September-October 2013• Duration : 6 months• Leader: MTA SZTAKI from the Hungarian NGI

• “EGI-InSPIRE is ideally placed to join together the new Distributed Computing Infrastructures (DCIs) such as clouds, supercomputing networks and desktop grids, for the benefit of user communities within the European Research Area.” --- EGI-InSPIRE Project fact sheet

• Motivation• Despite the on-going work and the benefits of the integrated Desktop Grid DCI

resources available for EGI users, only a part of the EGI community is aware of the latest Desktop Grid related achievements, and a fraction of the EGI users and infrastructure operators take their advantages in everyday practice.

• In order to fill this gap; a part of the targeted objectives are technical ones but there is stronger focus on the human aspects – training, networking and support activities.

• See list of Virtual Teams at https://wiki.egi.eu/wiki/Overview_of_Virtual_Team_projects

10

11

Proposed EGI DG-VT Project

• Promote and train the Desktop Grid related technologies in the EGI communities

• Utilize the available bridged Desktop Grid resources by more VOs

• Increase the number of heavily used EGI applications on the integrated (Desktop Grid) infrastructure with focus on widespread tools/solutions/approaches

• Improve documentation, e.g. Road maps, training materials, manuals

• Complete the final remaining steps for full integration concerning e.g. dashboard

• Find joint EGI champion(s) - IDGF ambassador(s)11

12

Current status of integration into the EGI infrastructure

https://wiki.egi.eu/wiki/Desktop_Grids_integration

12

13

EGI related activities and plans

• EGI activated its efforts in full integration of the Desktop Grid systems/projects to the EGI infrastructure and in promoting Desktop Grid among the EGI scientific users

• EGI plans to put manpower behind as well• Plans:

• EDGeS@home will be renamed to EGI@home• Technical discussion is needed, how to do that? Share your experiences

if any…• Screensavers for the EGI applications• Use the cooperation with BIOVEL and DIRAC as a positive

example• More dissemination (newsletters, leaflets) to notify scientific

users about DG resources13

14

Thank you for your attention…

Presenter:Jozsef Kovacs ([email protected])

Project website:http://idgf-sp.euhttp://desktopgridfederation.euhttp://www.lpds.sztaki.hu

Acknowledgements: The presented work is supported by the FP7 Capacities Programme

under contract nr RI-312297 (IDGF-SP).

14