The Earth System Grid (ESG)

The Earth System Grid (ESG)

Collaborationsand

Visibility

DOE SciDAC ESG Project ReviewArgonne National Laboratory, Illinois

May 8-9, 2003

May 8, 2003 Earth System Grid 2

Presentation Agenda

• Why Collaborate?

• Collaboration Framework and Partners

• Summary of ESG Collaborations

Part I

Why Collaborate?

May 8, 2003 Earth System Grid 4

Why Collaborate?

• Enterprise information sharing and management helps in productivity

• Coordination of effort to avoid the duplication of work (i.e., to avoid re-inventing the wheel, save money, time and resources)

• To bring together shared development of products useful to the entire community (e.g., Linux OS)

• Reach common Goals that will appeal to a wider audience

• It promotes visibility in the community and a wider acceptance of ESG components

• Promotes consistency and standards throughout the community

May 8, 2003 Earth System Grid 5

How can Collaborations Help?

• Keeping team members up to date• Promoting access and retrieval of data suitable

to user needs• Ensuring quality• Continuity• Minimizing biased code decisions and making

code more efficient and effective – Meaning looking out to the community for better ways

and avoid becoming insular

May 8, 2003 Earth System Grid 6

Goals of ESG Collaborations

• Leverage the work of others to meet ESG goals– Technology and operation experts and partner to

develop and maintain community components• Develop usable and re-usable products

– Usability means easy to learn, effective to use and provide an enjoyable experience for the user/developer

• Merge ESG technology with the external community by providing a wide-range of Grid-enabled tools

• Involve users in the design process– To make ESG more useful to climate researchers by

interacting with potential ESG users

May 8, 2003 Earth System Grid 7

Motivation and Strategies for ESG Collaboration

• Collaborate across teams and organizational boundaries with customers/partners

• Move outside the realm of ESG in decision making• Find the correct information quickly and easily

– work on it collaboratively, and – publish it centrally online for use and re-use

• Improve the inter-operability between the diverse national and inter-national groups and agencies

• Ensure that everyone on the project has up-to-date, reliable information and understands exactly what that information means to ESG and the community

May 8, 2003 Earth System Grid 8

ESG Outreach: Community Events and Interactions

• ECMWF Computing Workshop• Earth System Modeling Framework Workshop • NOMADS Workshops• High Performance Network Planning Workshop• SciDAC Visualization Workshop• Workshop on Data Provenance and Derivation• Kickoff Meeting, NCSA Alliance PACI Data Quest Expedition• National Institute for Environmental eScience (NIEeS) Launch Workshop• British Atmospheric Data Center (BADC)• NIEeS Workshop on Metadata and Ontology• Aceess Grid meetings with the SciDAC Climate group and the U.K. BADC

and CLRC teams• CODATA 2002 18th International Conference• DOE Townhall Meeting

Part II

Collaboration

Framework and Partners

May 8, 2003 Earth System Grid 10

ESG

AnalysisTools

Mgmt Services Data Portal

Web Services Arch. and Technologies

Climate ModelingResource Sharing and Provisioning

Storage andNetworking

Distributed Data

Mgmt Infrastructure

Globus GridTechnologies

• Disparate model• Coordinated

development processes• Shared technologies

Collaboration Framework

May 8, 2003 Earth System Grid 11

We leverage our own community, and the world.

Globus Toolkit Storage Resource Management

(SRM) Middleware Project

OPeNDAP (DODS): Distributed Oceanographic Data System (Unidata) Integrations of Globus GridFTP

THREDDS: framework for publishing, cataloging, describing scientific datasetsNcML - Co-development

CDAT: Climate Data Analysis Tools (PCMDI)

NCL: NCAR Command LanguageLAS: Live Access Server (NOAA Pacific

Marine Environmental Laboratory) Works with CDAT, NCL

Grid

Community

Climate

Community

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

May 8, 2003 Earth System Grid 12

Collaborations Partners

• CCSM Data Management Group• Other SciDAC Projects: Climate, Security & Policy for Group

Collaboration, Scientific Data Management ISIC, & High-performance DataGrid Toolkit

• CEOS-grid• e-Science, UK

– Council for the Central Laboratory of the Research Councils (CLRC)– Natural Environment Research Council (NERC) DataGrid

• Earth Science Portal (ESP) group (multi-agency, inter-national)• Earth System Modeling Framework (ESMF)• NOAA Operational Model Archive and Distribution System (NOMADS)• Center for Ocean-Land-Atmosphere (COLA) Studies

May 8, 2003 Earth System Grid 13

e-Science (CLRC and NDG): Collaboration Example

• ESG team and the e-Science team met via the AG for a collaboration work strategy team meeting– Common Goals– Interoperability between systems

• Proposted Tasks– Describe each group’s metadata– Overlaping areas– Fill in gaps in metadata– Merge CLRC, NDG, and ESG metadata where appropriate and

introduce higher-level abstract concepts between elements and build an ontology for the Earth Sciences

May 8, 2003 Earth System Grid 14

CDATClient

CDATClient

OPeNDAP: Leverage and Collaboration Example

• OPeNDAP Striped Server = OPeNDAPg

• OPeNDAPg Client Application Library

• Climate applications (e.g., NCL, CDAT) builds with OPeNDAPg Client Application Library

• Client Application Access GridFTP data via OPeNDAPg

NCLClient

NCLClient

Application Layer

Grid-enablednetCDF library

Grid-enablednetCDF library

OPeNDAPOPeNDAP

ncgrid clientncgrid client

Globus librariesGlobus libraries

OPeNDAPserver

OPeNDAPserver

striped GridFTPserver

striped GridFTPserver

Presentation Layer

Transport Layer

Data remoteData

OPeNDAP stripedserver module

OPeNDAP stripedserver module

Local?

On the Grid?

ESG Component Contributions

…

May 8, 2003 Earth System Grid 15

Collaboration Partnership Considerations

• Ownership of software and data– Open Source

• Security– Access to software and data– Authentication

• Funding the work that must be done– Maintenance of software and data repositories

• Liability– Software and data accuracy

May 8, 2003 Earth System Grid 16

Lessons Learned

• Embrace and leverage complementary activities• Keep the collaboration simple • Focus on core needs• Ensure a “win-win” collaborative effort• Facilitate constant feedback• Be open to giving all a chance• Don’t compete - facilitate• Meet the overall mission and purpose of ESG

Part II

Summary of ESG Collaborations

May 8, 2003 Earth System Grid 18

Summary of ESG Collaborations

• National and International collaboration• Develop methods, standards, and software

components integral to the success of ESG and usable to the larger Earth Science community

• Diverse internal structure (Metadata Groups, Grid and Network Groups, Data Portal Groups, Modeling Groups, and Data Visualization Analysis Groups)

• Provide unique contributions to the Earth Science communities

May 8, 2003 Earth System Grid 19

Summary of ESG Collaborations

• A Future challenge will be to maintain efficient and effective communication between all ESG collaborations.