The Future of EGEE and gLite

EGEE-II INFSO-RI-031688

Enabling Grids for E-sciencE

www.eu-egee.org

The Future of EGEE and gLite

Dieter KranzlmüllerGUP – Institute of Graphics and Parallel ProcessingJoh. Kepler Univ. Linz, Austria

CE EGEE&SEEGRID-2 Summer School on Grid Appl.

08 July 2006Budapest, Hungary

The Future of EGEE and gLite 2



Austrian Grid Initiative

GGF - Global Grid Forum

e-IRG - e-Infrastructure Reflection Group

EGEE – Enabling Grids for E-sciencECERN, Geneva, Switzerland

GUP – Institute of Graphics and Parallel ProcessingJoh. Kepler University Linz




Lost in Definitions?

Defining the “Grid”:• Access to (high performance) computing power• Distributed parallel computing• Improved resource utilization through resource sharing• Increased memory provision• Controlled access to distributed memory• Interconnection of arbitrary resources

(sensors, instruments, …)• Collaboration between users/resources• Higher abstraction layer above network services• Corresponding security • …




Defining the Grid

• A Grid is the combination of networked resources and the corresponding Grid middleware, which provides Grid services for the user.

• This interconnection of users, resources, and services for jointly addressing dedicated tasks is called a virtual organization.

• Comparison between Grids and Networks:– Networks realize message exchange between endpoints– Grids realize services for the users higher level of abstraction




Defining the Grid





The EGEE Project

• EGEE– 1 April 2004 – 31 March 2006– 71 partners in 27 countries,

federated in regional Grids

• EGEE-II– 1 April 2006 – 31 March 2008– Expanded consortium

91 partners 11 Joint Research Units (48 partners)

– Exploitation of EGEE results– Emphasis on providing

production-level infrastructure increased support for applications interoperation with other

Grid infrastructures more involvement from Industry




Defining the Grid


Status of EGEE-II (as of July 8, 2006)




EGEE Infrastructure

Country participating

in EGEE

Scale (June 2006):~ 200 sites in 40 countries

~ 20 000 CPUs

> 10 PB storage

> 20 000 concurrent jobs per day

> 60 Virtual Organizations




EGEE Infrastructures

• Production service– Scaling up the infrastructure with resource centres around the globe– Stable, well-supported infrastructure, running only well-tested and

reliable middleware

• Pre-production service– Run in parallel with the production service (restricted nr of sites)– First deployment of new versions of the gLite middleware – Test-bed for applications and other external functionality

• T-Infrastructure (Training&Education)– Complete suite of Grid elements

and application (Testbed, CA, VO, monitoring, …)

– Everyone can register and use GILDA for training and testing

15 sites on 3 continents(all of them GÉANT sites)




Defining the Grid






Production Grid Middleware

Key factors in EGEE Grid Middleware Development:

1. Strict software processUse industry standard software engineering methods– Software configuration management, version control, defect

tracking, automatic build system, …

2. Conservative approach in what software to useAvoid “cutting-edge” software– Deployment on over 100 sites cannot assume a homogenous

environment – middleware needs to work with many underlying software flavors

Avoid evolving standards– Evolving standards change quickly (and sometime significantly

cf. OGSI vs. WSRF) – impossible to keep pace on > 100 sites

Long (and te

dious) path

from prototypes to

production




EGEE Middleware: gLite

• Exploit experience & existing components – VDT (Condor, Globus) – EDG/LCG– AliEn– …

• Develop a lightweight stack of EGEE generic middleware– Dynamic deployment– Pluggable components

• Focus is on re-engineering and hardening

• March 4, 2006: gLite 3.0

LCG-2

prototyping

prototyping

product

2004200420042004

20052005 product

gLite

20062006 gLite 3.0




Developing

• gLite 3.0 now available on production infrastructure

• After gLite 3.0:– Continuous release of single components

As needed by users and as made available by developers

– Major releases provide a “check-point” In general in coincidence with major application challenges

• Continuing development to– Bring components not yet included in release to maturity – Improve functionality– Increase robustness– Increase usability– Improve the compliance to international standards




Grid Interoperability

Leading role in building world-wide grids• Incubator for new Grid

projects world-wide

• Interoperation efforts– Bilateral: EGEE/OSG, EGEE/NDGF,

EGEE/NAREGI– Multilateral: Grid Interoperability Now

(GIN)

• Experiences and requirements fed back into standardization process (GGF)

• Strengthening contacts with industry

GIN




MiddlewareGlobus GT4 CondorAPST

PlatformInfrastructure

Unix Windows JVM TCP/IP MPI .Net Runtime

Environmental Sciences

Life & Pharmaceutical

Sciences

ApplicationsGeo Sciences

Building Software for the Grid

VPN SSH

Courtesy IBM

Slide Courtesy David Abramson









Sciences



VPN SSH

Courtesy IBM,Lower Middleware

Upper Middleware & Tools

Bonds





Defining the Grid






EGEE Applications

• >20 applications– High Energy Physics– Biomedicine– Earth Sciences – Computational Chemistry– Astronomy– Geo-Physics– Financial Simulation– Fusion

• Further applications in evaluation

Applications now moving from testing to routine and daily usage




High Energy Physics

Large Hadron Collider (LHC):• One of the most powerful instruments

ever built to investigate matter• 4 Experiments: ALICE, ATLAS, CMS, LHCb • 27 km circumference tunnel• Due to start up in 2007

Mont Blanc(4810 m)

Downtown Geneva




Applications Example: WISDOM

• Grid-enabled drug discovery process for neglected diseases– In silico docking

compute probability that potential drugs dock with target protein

– To speed up and reduce cost to develop new drugs

• WISDOM (World-wide In Silico Docking On Malaria)– First biomedical data challenge – 46 million ligands docked in 6 weeks

Target proteins from malaria parasite Molecular docking applications:

Autodock and FlexX ~1 million virtual ligands selected

– 1TB of data produced – 1000 computers in 15 countries

Equivalent to 80 CPU years

• Significant results– Best hits to be re-ranked using Molecular Dynamics

New data challenge planned

for a

utumn 2006




Example: Avian flu

• Avian Flu H5N1– H5 and N1 = proteins on virus surface

• Biological goal of data challenge– Study in silico the impact of selected point

mutations on the efficiency of existing drugs – Find new potential drugs

• Data challenge parameters:– 5 Grid projects: Auvergrid, BioinfoGrid, EGEE,

Embrace, TWGrid– 1 docking software: autodock– 8 conformations of the target (N1)– 300 000 selected compounds >100 CPU years to dock all configurations

on all compounds

• Timescale: – First contacts established: 1 March 2006– Data Challenge kick-off: 1 April 2006– Duration: 4 weeks

N1H5

Credit: Y-T Wu

Credit: Y-T Wu




Example: Determining earthquake mechanisms

• Seismic software application determines epicentre, magnitude, mechanism

• Analysis of Indonesian earthquake (28 March 2005)– Seismic data within 12 hours after the earthquake– Solution found within 30 hours after earthquake occurred

10 times faster on the Grid than on local computers– Results

Not an aftershock of December 2004 earthquake Different location (different part of fault line further south) Different mechanism

Rapid analysis of earthquakes important for relief efforts

Peru, June 23, 2001Mw=8.4

Sumatra, March 28, 2005Mw=8.5




EGEE-II Overview




EGEE and Sustainability

BUT …

• How does EGEE compare to other computing infrastructures?– Number of infrastructure users?– Number of application domains?– Number of computing nodes?– Number of years in service?

• What would happen, if EGEE is turned off?• What happens after April 2008

(End of EGEE-II)?




The Future of Grids

• Increasing the number of infrastructure users by increasing awareness– Dissemination and outreach– Training and education

• Increasing the number of applications by improving application support and middleware functionality– High level grid middleware extensions

• Increasing the grid infrastructure– Incubating related projects– Ensuring interoperability between projects

• Protecting user investments– Towards a sustainable grid infrastructure




User Information & Support

• More than 170 training events and summer schools across many countries– >3000 people trained

induction; application developer; advanced; retreats– Material archive online with ~250 presentations

• Public and technical websites • Dissemination material constantly evolving to expand

information and keep it up to date

• 4 conferences organized (~ 460 @ Pisa)

• Next conference: September 2006 in Geneva ~600 participants




Industry and EGEE-II

• Industry Task Force– Group of industry partners in the project– Links related industry projects (NESSI, BEinGRID, …)– Works with EGEE’s Technical Coordination Group

• Collaboration with CERN openlab project– IT industry partnerships for hardware and software

development

• EGEE Business Associates (EBA)– Companies sponsoring work on joint-interest subjects

• Industry Forum– Led by Industry to improve Grid take-up in Industry– Organises industry events and disseminates grid information




The Future of Grids













Sciences



VPN SSH



Bonds


???




Portals on EGEE

P-Grade

Genius




www.austriangrid.at

High Level Middleware Extensions

Example:• Understanding data through graphical representations

Scientific Visualization

• Approach: Grid Visualization Kernel (GVK)– Interactive visualization service in the grid– Integrated in existing visualization toolkits– Optimized transportation and processing

• Research project:– Interactivity on the Grid glogin– Batch-bypass technology– Easy to use/install– Secure




glogin - Interactive Tunneling

WorkerNode

WorkerNode

WorkerNode

WorkerNode

WorkerNode

WorkerNode

WorkerNode

WorkerNode

WorkerNode

WorkerNode

GatekeeperGatekeeper

on the Grid

ClientClient

gloginPoint ofContact

glogin’socket

InteractiveBidirectional Connection

TrafficForwarding




glogin Shell

• Interactive access to grid nodes

• Authentication via grid certificates• Tunneling of arbitary traffic




Grid Visualization Kernel

Workernode

Workernode

Workernode

Workernode

Workernode

Workernode

Workernode

Workernode

GridVisualization

Kernel

GridVisualization

Kernel

on the Grid

ClientClient

glogin

glogin’

vis.data

vis.data

vis.data W

OR

KIN

G!

inter-action

interaction interaction




Example: Biomedicine

• Parallel simulationof blood flowon the Grid

• Onlinevisualizationof simulationresults on thedesktop

• Interactivesteering ofsimulation

• Grid is„invisible“

Cooperation with University Amsterdam




Example: Flooding Crisis Support

• Simulation of floodingon the Grid

• Onlinevisualizationof simulationresults in theCAVE

• Interactivesteering ofsimulation

• Grid is„invisible“

Cooperation with Slowak Academy of Sciences




Scientific Visualization

Use your favourite device to connect to the Grid:

Sony PSP – PlayStation Portable









Sciences



VPN SSH



Bonds


GENIUS, P-Grade, GVK, glogin, GVid, …




The Future of Grids








Projects related to EGEE

EUGRIDGRID




Related Infrastructures

GIN




The Future of Grids








Sustainability: Beyond EGEE-II

• Need to prepare for permanent Grid infrastructure– Maintain Europe’s leading position in global science Grids– Ensure a reliable and adaptive support for all sciences– Independent of project funding cycles– Modelled on success of GÉANT

Infrastructure managed centrally in collaboration with national bodies (in EGEE-II: JRUs)




Grids in Europe

• Great investment in developing Grid technology• Sample of National Grid projects:

– Austrian Grid Initiative– DutchGrid – France: Grid’5000– Germany: D-Grid; Unicore– Greece: HellasGrid– Grid Ireland – Italy: INFNGrid; GRID.IT– NorduGrid– Swiss Grid– UK e-Science: National Grid Service;

OMII; GridPP

• EGEE provides framework for national, regional and thematic Grids




Evolution

EGEE EGEE-IIEDG EGEE-III

European e-Infrastructure

Coordination

Testbeds Utility ServiceRoutine Usage




Summary

Grids represent a powerful new tool for science

Today we have a window of opportunity to move grids from research prototypes to permanent production systems (as networks did a few years ago)

EGEE offers …• … a mechanism for linking together people, resources

and data of many scientific community• … a basic set of middleware for gridfying applications with

documentation, training and support• … regular forums for linking with grid experts, other

communities and industry




Summary

Success will lead to the adoption of grids as the main computing infrastructure for science

If we succeed then the potential return to international scientific communities will be enormous and open the path for commercial and industrial applications




EGEE’06 Conference

• EGEE’06 – Capitalising on e-infrastructures – Demos– Related Projects– Industry– International community (UN organisations in Geneva etc.)

• 25-29 September 2006• Geneva, Switzerland

http://www.eu-egee.org/egee06

The Future of EGEE and gLite

Documents