NA4: User Community Support and Expansion

Slide 1

C. Loomis (CNRS/LAL)NA4 Activity Manager

EGEE-III Final Review, 23-24 June 2010NA4: User CommunitySupport and ExpansionEGEE-III INFSO-RI-222667 Enabling Grids for E-sciencEwww.eu-egee.orgEGEE and gLite are registered trademarks Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 1

Activity OverviewNA4 - C. Loomis - EGEE-III Final Review 23-24 June 20102CountryPMFTEAustria60.3Belgium120.5CERN1626.8Cyprus120.5Czech Republic170.7France25710.7Germany472.0Greece923.8Hungary1265.3Israel120.5Italy2098.7Netherlands291.2Norway301.3Poland421.8Russia662.8Slovakia241.0Spain2118.8Sweden160.7UK411.7TOTAL142159.244Partners287People19CountriesNA4: 19%Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 2TasksTNA4.1: SupportVirtual Organization Support (VOS)Application Porting Support (APS)Direct User Support (DUS)TNA4.2: Strategic Discipline ClustersHigh Energy Physics (HEP)Life Sciences (LS)Earth Sciences (ES)Grid Observatory (GO, CS)Computational Chemistry (CC)Astronomy & Astrophysics (AA)Fusion (F)TNA4.3: Activity CoordinationActivity ManagementRegional CoordinationNA4 - C. Loomis - EGEE-III Final Review 23-24 June 20103Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 3CPU Utilization Over TimeNA4 - C. Loomis - EGEE-III Final Review 23-24 June 20104Accounting Portal: http://www3.egee.cesga.es/ CIC Portal: http://cic.gridops.org/

EGEE-III = Y1EGEE-II = Y0EGEE-III = Y2Consistent doubling every 12-18 months.Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 4CPU Utilization by DisciplineNA4 - C. Loomis - EGEE-III Final Review 23-24 June 20105DomainY0Y0 (%)Y1Y1 (%)Y2Y2 (%)Y1/Y0Y2/Y1AA18550.8%151153.1%121551.4%8.10.8CC126545.8%351127.2%82391.0%2.80.2CS00.0%530.0%20.0%ES3180.1%26750.6%25320.3%8.40.9F22261.0%24460.5%45890.5%1.11.9HEP17785781.0%37691077.6%78403590.7%2.12.1INF33341.5%13370.3%18580.2%0.41.4LS106504.8%206094.2%153731.8%1.90.7MV57392.6%119292.5%88491.0%2.10.7OTH20150.9%135572.8%199312.3%6.71.5UNK30461.4%59381.2%64550.7%1.91.1TOTAL219694100.0%485680100.0%864018100.0%2.21.8Non-HEP: 2.6 (Y1/Y0), 0.7 (Y2/Y1)Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Applications and UsersNA4 - C. Loomis - EGEE-III Final Review 23-24 June 20106DisciplineApps.AA55CC17CS30ES19F6HEP13LS55OTH53TOTAL248RegionApps.Asia/US3Benelux7Central Europe6CERN/DE/CH7FR/UK/IRE21IT154Northern Europe10Southeast Europe30Southwest Europe10TOTAL248DomainVOsUsersAA21348CC4439CS625ES10267F280HEP434859*LS13561MV291639OTH361782TOTAL16410000Total in Y1: 9540Alternate link: http://grid.ct.infn.it/egee_applications/ Application Database: http://appdb.eu-egee.org

Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Active Virtual OrganizationsNA4 - C. Loomis - EGEE-III Final Review 23-24 June 20107Y1Y2DomainVOs(>0%)VOs(>10%)VO Names(>10%)VOs(>0%)VOs(>10%)VO Names(>10%)AA113astro.vo.eu-egee.org, auger, virgo135argo, auger, glast.org, icecube, virgoCC42compchem, trgrida42compchem, trgridaCS21imath.cesga.es21vo.mosfet.es-ngi.euES51esr52egeode, esrF11fusion11fusionHEP294alice, atlas, cms, lhcb344alice, atlas, cms, lhcbINF202euindia, nw_ru184hungrid, nw_ru, ops, vo.rhone-alpes.idgrilles.frLS81biomed83biomed, emnr.eu, lsgridMV163balticgrid, seegrid, vo.gear.cern.ch153balticgrid, see, vo.northgrid.ac.ukOTH192geant4, theophys231theophysUNK582litgrid, vo.nanocmos.ac.uk523litgrid, vo.nanocmos.ac.uk, vo.ssp.ac.ukEnabling Grids for E-sciencEEGEE-III INFSO-RI-222667 22 heavy VOs in Y1 (173 total)29 heavy VOs in Y2 (175 total)7Virtual Organization SupportVO monitoring servicesThese are based on the standard monitoring tools of the EGEE infrastructure and will allow VOs to exclude sites that do not pass VO-specific tests, thereby reducing job failures. This is being prototyped with the biomed VO and will be extended to others after validation.VOMS-enabled collaborative toolsThis system extracts membership and group information from the VOs VOMS server and uses that information to configure various collaborative tools like mailing lists, chat servers, and secured web sites. VO software deployment toolsDeveloped originally for the atlas VO, LJSFI could be used by many other VOs with few modifications. Instructions for using it have been written and are available on the teams wiki page.CIC PortalThis is a core service used in the VO registration and management. The group contributed to the evolution of this service in order to make it as pertinent as possible for all of the actors in EGEE.Documentation and Support ProvisionLinks to VO documentation: https://twiki.cern.ch/twiki/bin/view/EGEE/VOSupport VO documentation: The group has maintained on its wiki site a FAQ for typical VO manager questions and links to other documentation of interest to VO managers.NA4 - C. Loomis - EGEE-III Final Review 23-24 June 20108Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 8Application Porting SupportConsultancy and PortingOver the last year group has helped port around 20 applications to the EGEE grid infrastructure. These have resulted in 7 papers with more expected as those applications mature.Use of RESPECT ToolsThe team is well versed in the utilization of the RESPECT packages that complement the functionality of gLite. Because of this the team members have participated in many of the training events organized by the NA3 activity.Application DatabaseThis database keeps track of applications running on the EGEE infrastructure and provides a good view of the impact that the EGEE infrastructure has had on the user communities. This database was a cooperative effort of the Application Porting Support Team and the Greek Regional Coordination personnel.MPI ImprovementsMPI is important for a large number of scientific domains. The group has participated strongly in the MPI Working Group to improve the support and efficiency of MPI on the EGEE grid infrastructure.Case StudiesFor each ported application, a case study is written. This both documents the work of the group and provides information for porting similar applications. All of the case studies are available on the teams web pages.NA4 - C. Loomis - EGEE-III Final Review 23-24 June 20109Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 9Direct User SupportTicket HandlingThe group created a support group within the GGUS system and operated 2-person, 2-week shifts to handle tickets. Tickets were generally handled quickly, although few tickets were assigned to this support group.Use CasesGeneral use cases were updated and maintained, although the main focus was on creating specific use cases for each scientific area. A number of new use cases were generated, with those from chemistry and astrophysics being the most popular.ScreencastsIn order to make the documentation more relevant and more appealing to users a set of 12 screencasts were created based on the existing generic use cases. Those are now available and will hopefully improve the user support experience.NA4 - C. Loomis - EGEE-III Final Review 23-24 June 201010Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 10Large investment in testing and development of high-level tools validated by the smooth, successful start of LHC data taking and analysis.Developed tools (AMGA, Dashboard, Ganga/Diane) are heavily used within other scientifici communities.High Energy PhysicsNA4 - C. Loomis - EGEE-III Final Review 23-24 June 201011

Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 11Life SciencesConcentrated on adapting and improving tools for the life science communities: medical imaging, bioinformatics, and drug discovery.Tools include AMGA and encrypted data management.Goal: maximize grid utility while simplifying access.NA4 - C. Loomis - EGEE-III Final Review 23-24 June 201012

Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 12Earth SciencesCommunity building is the core strategy.Done by propagating grid technology to all ES disciplines, setting up interactive collaboration among the members of the virtual ES grid community, and stimulating the interest of stakeholders on the political level.Large increase in users within this discipline. NA4 - C. Loomis - EGEE-III Final Review 23-24 June 201013

Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 13Grid ObservatoryThe number of users has doubled this last year, with a broad geographic distribution, and steadily increasing usage of the portal facilities.The GO usage has started to live its own life, with usage that has extended well beyond the cluster partners. NA4 - C. Loomis - EGEE-III Final Review 23-24 June 201014

Grid Observatory: http://www.grid-observatory.org/ Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 14Regional DistributionNA4 - C. Loomis - EGEE-III Final Review 23-24 June 201015

Grid Observatory: http://www.grid-observatory.org/ Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 15Computational ChemistryCreation of a customized portal for simplified access to the grid.Adaptation of applications to treat chemical interactions in new regimes that would be difficult or impossible without the grid.Expansion and parallelization of standard chemical software packages for use on the grid. NA4 - C. Loomis - EGEE-III Final Review 23-24 June 201016

Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 16Astronomy & AstrophysicsPorting of new applications to the grid infrastructure.Provision of tools to improve the utility of the grid for its scientists.Interfaces to important data stores outside of the grid.Training of new users. NA4 - C. Loomis - EGEE-III Final Review 23-24 June 201017

Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 17FusionImproved collaboration between fusion institutes, projects, and infrastructures.For the first time, a complex workflow between EGEE-III, EUFORIA, and DEISA infrastructures has been carried out (see demo).NA4 - C. Loomis - EGEE-III Final Review 23-24 June 201018

Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 18Activity CoordinationActivity ManagementAll milestones and deliverables have been achievedMaintained and expanded the RESPECT programEncouraged community interaction via the User ForumContributed to EGEE EGI migration (see following)

Regional CoordinationDesign, implementation, and filling of Application DatabaseDeveloped knowledge of user communities within regionsNA4 - C. Loomis - EGEE-III Final Review 23-24 June 201019Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 19RESPECTIdentify third-party software that works well with gLite.http://technical.eu-egee.org/index.php?id=290 Integrated with application database in EGI

Y1 Tools (11)P-GRADE, Ganga, Migrating Desktop, g-Eclipse, i2glogin, Virtual Control Room, GridWay Metascheduler, DIANE, GRelC, Instrument Element, StoRMY2 Tools (7)ROOT/PROOF, VBrowser, Dashboard, DiGS, Work Binder Application Service, ToPoS, ESIPNA4 - C. Loomis - EGEE-III Final Review 23-24 June 201020Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 11 in Y118 in Y220User ForumNA4 - C. Loomis - EGEE-III Final Review 23-24 June 201021http://technical.eu-egee.org/index.php?id=148

UF1(CERN)UF2-OGF20(Manchester)UF3(Clermont-Ferrand)UF4-OGF25(Catania)UF5(Uppsala)

electronicJOGC (V8, No.2, June 2010): http://www.springerlink.com/content/g6234w046001/Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 21Transition EGEE EGISeries of SSC workshops:Used to define user community model in EGIUsed to define strategy for VRC organization and supportVirtual Research CommunitiesDefined needs to move towards sustainabilityBroadened and strengthened ties within existing communitiesNew communities: Humanities, Photon, and Complexity Science

Result:EC project proposals: ROSCOE, SAFE, CUE, SGI, TAPAS, None passed initial quality thresholdsLimited funding of existing communities through EGI-InSPIRENA4 - C. Loomis - EGEE-III Final Review 23-24 June 201022Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 22Support Activities After EGEEVO SupportExisting documentation available: https://twiki.cern.ch/twiki/bin/view/EGEE/VOSupport Recommendations for VO management: https://edms.cern.ch/document/1070320/1CIC Portal will continue as part of EGI operations tools Application Porting SupportExisting team and portal will continue in EGI-eraExpertise conserved within the EGI communityDirect User SupportUse cases and screencasts available for use/evolutionCurrent ticket handling team will not continueDifficult to transfer expertise as responsibility is very distributedNA4 - C. Loomis - EGEE-III Final Review 23-24 June 201023Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 Virtual Research CommunitiesFailure of EC-funded proposals:Raised serious doubts about VRC model itselfLeft (most) existing communities without funded effort for European-level coordination of grid-related activitiesBeginning of EGI:Need to review/validate user community organizationNeed to review/validate interaction model with EGICommunities will continue on best-effort basis in short termConcerns:Lack of European-level coordination may lead to fragmentation of communitiesReliance on ESFRI projects alone will ignore large segments of scientific communitiesNA4 - C. Loomis - EGEE-III Final Review 23-24 June 201024Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 SummaryUser Community10000 users, 248 applications, 175 VOsMajority of use from 29 core VOsOverall CPU 2x greater, driven by HEP usageImpactDNA4.5 contains details on technical and scientific impactBooks of Abstracts and special JOGC issue show breadth EGEE EGI TransitionCommunities worked hard for realistic, viable transitionCurrent situation concerns most communitiesNA4 - C. Loomis - EGEE-III Final Review 23-24 June 201025Enabling Grids for E-sciencEEGEE-III INFSO-RI-222667 25