OMII-UK Campus Grid Toolkit

Web: www.omii.ac.uk Email: [email protected]

OMII-UK Campus Grid Toolkit

NW-GRID Campus Grids Workshop31st October 2007

University of LiverpoolTim Parkinson

OMII-UK Southampton Operations Manager


OMII-UK

• What is OMII-UK?o Collaboration between Open Source Software

developers at Southampton, Manchester, Edinburgh.

• Mission:o OMII-UK aims to provide software and support to

enable a sustained future for the UK e-Science community and its international collaborators.

• Software Solutions for e-Research• Led by user / community requirements


Characteristics of a Campus Grid

Generally Speaking....• One or more

o HPC Clusterso Condor Poolso Specialised Computation Devices or Data

Resourceo Owned and managed by a single institutiono Subject to single AuthN regime and internal

AuthZ policies?o Shared by users belonging to the institutiono Applications / Services useful to userso Maintenance & support of that environment


Uses of a Campus Grid

• Typicallyo To run large scale computations or

simulationso That can be partitioned into scenarios or

parameter sweeps.o That therefore benefit by running many

scenarios in parallel or on larger machines.• Reduce Time to Publication• Obtain results that would be unobtainable in

available time if done sequentially or on a smaller desktop.


Requirements of Campus Grid Users

• Determined from a survey of the UK Campus Grid SIG membership

o Chaired by David Wallom, OeRC

• Backed up by the SUPER reporto Study of User Priorities for e-

Infrastructure for e-Research (NeSC 2007)

o http://www.nesc.ac.uk/technical_papers/UKeS-2007-01.pdf


SIG Requirements

• Five areas: Applications

How to transparently execute applications across heterogeneous resources to maximise throughput and minimise execution time.

Identify common applications for shrink-wrapping.

SecurityUsers: Just want to login once and use what they are

allowed to use transparently or to be refused gracefully.Admins: Want to be able to apply suitable authentication

method and suitable access control policy. Want to support attribute based virtual organisations (VOMS / Shibboleth) to support collaborations.


SIG Requirements

• Five areas (ctd)• Accounting:

o Admins: Driven by FEC. Associate usage with user. Ability to log, price, and ultimately to bill.

o Users: Any such billing should be fair and trustworthy.

• Monitoringo Admins: Track resource utilisation and

availabilityo Users: See what is happening to their jobs.


SIG Requirements

• Five areas (ctd)• Storage

o Users: ability to seamlessly transfer data to and from a variety of distributed storage systems (such as SRB) or databases.

o Admins: ability to configure and such storage mechanisms into job services and to monitor them.


SUPER Requirements

• Similar to CG SIG• Spectrum of User Interaction Styles

o Web Portals Wrapped Applications

o Desktop GUIo CLIo Scripting Languages o Programmatic API Custom

Applications


Quality Requirements

• Installabilityo Ease of installation and configuration.

• Reliabilityo Should stay up and running or at least fail

gracefully

• Portability / Availabilityo Should work on a range of different architectures

and operating systems and back end job managers.

• Scalabilityo Should work for personal installations up to

Campus Wide and beyond.


What is the Campus Grid Toolkit?

• An enhanced packaging of existing and future OMII-UK components, principally GridSAM, that provides

o Consistent job submission across heterogeneous resources for the scientist (via OGF standard JSDL) over a Web Services interface.

o Ease of installation and configuration for the administrator.


What is the Campus Grid Toolkit

• Provides (ctd)o a range of interaction styles for the scientist

(Desktop, CLI, Portal / Portlets)o a range of configurable security policies for

the administrator (OMII-AuthZ, SPAM-GP, VOMS / Shibboleth integration).

o a way to wrap legacy, unmodified applications (AHE, OGRSH)

o a way to create new applications that access grid resources directly (SAGA and its scripting bindings)


What is the Campus Grid Toolkit?

• A vehicle to deliver an integrated set of OMII-UK components that work together to enhance the scientist’s ability to submit large numbers of computational jobs to the resources available on campus and beyond in a seamless fashion.

• Should become the installation of choice for Campus Grid providers.


First Release (Nov 2007)

• Address the Installability of GridSAM onto a Condor pool.

• GridSAM on Condor in a Box.• Autoconfiguration• Better example programs• Address reliablity issues• Prototype demonstrated at OGF21


The Old Way – Component Based

• Follow an installation and configuration process (1-2 hours):

1. Install the OMII-Server bundle and select GridSAM as an option

2. Use temporary server certificate3. Configure the Condor DRM yourself4. Download and install the client5. Use temporary client certificate6. Test with uname application (too trivial)7. Make client available to users8. Replace temporary CA and cert with real ones


The New Way

• Move towards one-step installation and configuration (10 mins):

1. Install CGT which offers the option to trust UK CA and configures server accordingly

2. Attempts to install trust for UK CA and your own ‘real’ certificate

3. Attempts to detect and auto-configure Condor DRM

4. Automatically installs and configures client to match the server that was just installed

5. Test with Mandelbrot set – a real application requiring significant processing time

6. Make pre-configured client available


Future Plans (no particular order)

• Add one-step installation and auto-configure for AHE (Demo SC2007)

• Extend one-step installation to other DRMs (Globus, PBS/Torque, Platform LSF etc)

• Extend GridSAM to propagate user identity and to collect resource usage for jobs.

• Investigate ways to add grid monitoring, perhaps with OGM

• Investigate job service brokering perhaps with Knoogle or Grid-BS.


Future Development

• Enhance all major OMII-UK components to interact with the main storage management solutions.


• Questions?

OMII-UK Campus Grid Toolkit

Documents