May 6, 2002Earth System Grid - Williams The Earth System Grid Presented by Dean N. Williams PI’s: Ian Foster (ANL); Don Middleton (NCAR); and Dean Williams.

May 6, 2002 Earth System Grid - Williams

The Earth System GridPresented by

Dean N. Williams

PI’s: Ian Foster (ANL); Don Middleton (NCAR); and Dean Williams (LLNL)

http://www.earthsystemgrid.org

Presented at: The “EO GRID” Workshop

Frascati, Italy

UCRL-PRES-148116

May 6, 2002 Earth System Grid - Williams

Funded by the Scientific Discovery through Advanced Computing (SciDAC), this program seeks a new paradigm in the climate change community evolving from centralized data sharing to distributed data-sharing.

Enabling geographically distributed teams of researchers to effectively and rapidly acquire knowledge and understanding of massive amounts of climate data holdings.

Multiple interfaces to ESG will allow researchers to focus on science and not issues with data receipt, format, and data set manipulation.

Earth System Grid (ESG): Overview

May 6, 2002 Earth System Grid - Williams

ESG: Why is ESG Important to the U.S. Climate Change Program

Climate model output and quality observations are vital to providing timely assessments of climate change and impacts.

Recent U.S. and IPCC assessment efforts made it clear the lack of accessibility to model simulations is a major problem for future assessments.

Access to retrospective climate data (input and output) needed to enable a feedback mechanism to tie researchers directly back to quality control and diagnostics of models.

Researchers require access to “format independent” climate and observational data for case-study & training.

In the U.S., climate simulation can be viewed as a systems problem, requiring a team of multi-agencies and institutions working together in collaboration.

May 6, 2002 Earth System Grid - Williams

ESG: U.S. Collaborations & Development

ORNL: Climate storage &computational resources

ORNL: Climate storage &computational resources

LANL: Next generationcoupled models & computing

LANL: Next generationcoupled models & computing

ANL: Computational grids,& grid-based applications

ANL: Computational grids,& grid-based applications

USC/ISI: Computational grids,& grid-based applications

USC/ISI: Computational grids,& grid-based applications

NCAR: Climate changepredication and scenarios

NCAR: Climate changepredication and scenarios

LBNL: Climate storage facility

LBNL: Climate storage facility

LLNL: Model diagnostics& inter-comparison

LLNL: Model diagnostics& inter-comparison

May 6, 2002 Earth System Grid - Williams

ESG: Requirements & Priority Matrix

ESG Developer ESG Administrator ESG UserESG Services: Framework H H H Automatic Installation L L HDistributed Computing Authorization & AuthenticationH H M Registration H H L Event Services L L M Task Management L L L Logging Services L H HData Systems Search and Discovery M H H data movement (transport) L H H meta-data framework H H M collaboratories M L HTools analysis M M H visualization L L H collaboration M M H

L = LOW, M = MEDIUM, H = HIGH

May 6, 2002 Earth System Grid - Williams

ESG: U.S. Department of Energy (DOE) Next Generation Internet

(NGI) Project ESG-I (past):

Focused on developing techniques for the high-speed data movement between sites and users (e.g., the secure highly efficient File Transfer service, called gridFTP, developed by ANL (i.e., Globus))

Developed replica catalogs for keeping track of data locations

Developed request manages for coordinating multiple transfers

Developed a grid-enabled version of LLNL’s data analysis package

May 6, 2002 Earth System Grid - Williams

ESG: ESG-I Architecture

May 6, 2002 Earth System Grid - Williams

ESG: ESG-I Team Presented their work at Supercomputing 2001

parallel disk system

LDAP/SeverMetadataCatalog

ANL

tape system parallel disk system

Network

LDAP/SeverMetadataCatalogLLNL

LDAP/SeverMetadataCatalogLBNL

tape system

LDAP/ServerMetadataCatalogSC ‘01

RAID

LocalDisks

CLOUD

TERRAIN U & V

May 6, 2002 Earth System Grid - Williams

ESG: DOE SciDAC Project

ESG-II (present): Building upon the substantial work of ESG-I Grid-wide services supporting authentication, authorization, data

discovery, and user specified analysis Metadata services supporting remote data browsing, querying,

accessing, displaying, etc. Filtering services performing intelligent model specific analysis

before delivering the results to the user Integrate next-generation data analysis and visualization

applications (such as ongoing work at LLNL and NCAR), web-based data portals and other thin clients supporting the Distributed Oceanographic Data System (DODS), and collaborative problem-solving environments.

May 6, 2002 Earth System Grid - Williams

ESG: ESG-II Architecture

May 6, 2002 Earth System Grid - Williams

ESG: Metadata Services

METADATAEXTRACTION

METADATAEXTRACTION

METADATADISPLAY

METADATADISPLAY

METADATABROWSING

METADATABROWSING

METADATAQUERY

METADATAQUERY

ESG CLIENTS API & USER INTERFACES

Data &MetadataCatalog

Dublin CoreDatabase

COARDSDatabase

mirrorDublin CoreXML Files

COMMENTSXML Files

METADATA HOLDINGS

METADATAANNOTATION

METADATAANNOTATION

METADATAVALIDATION

METADATAVALIDATION

METADATA ACCESS(update, insert, delete, query)

METADATA ACCESS(update, insert, delete, query)

SERVICE TRANSLATIONLIBRARY

SERVICE TRANSLATIONLIBRARY

CORE METADATA SERVICES

METADATAAGGREGATION

METADATAAGGREGATION

METADATADISCOVERY

METADATADISCOVERY

METADATA & DATA REGISTRATION

METADATA & DATA REGISTRATION

PUBLISHINGPUBLISHING

HIGH LEVEL METADATA SERVICES

SEARCH & DISCOVERYSEARCH & DISCOVERYADMINISTRATIONADMINISTRATION BROWSING & DISPLAYBROWSING & DISPLAY

ANALYSIS & VISUALIZATIONANALYSIS & VISUALIZATION

May 6, 2002 Earth System Grid - Williams

Grid and NetworkInfrastructure

Onlinestorage systems

Computationalresources

? RCAS

ESG services: information, replica,metadata, community authorization

M

Data consumers

Data producers

ESG: Collaboration Network

May 6, 2002 Earth System Grid - Williams

ESG: Example of a Web-based Data Portal (currently serving 40+ simulations of AMIP, CMIP, and PCM data, and growing)

May 6, 2002 Earth System Grid - Williams

ESG: Example of a Client Application

May 6, 2002 Earth System Grid - Williams

ESG: Example of a Script Access

The next-generation language, Python, is used to access the Earth System Grid at LLNL

Import cdms

db = cdms.open(“ldap://localhost:389/database=demo,ou=PCMDI,o=LLNL,c=US”)f = db.open( “ncep_reanalysis_mo”)ds = f(‘ts’)

May 6, 2002 Earth System Grid - Williams

ESG: Concluding Statements ESG is a highly collaborative effort and will allow users to quickly

access data storage facilities storing petabytes of raw or processed data in an application independent manner.

Payoffs of this distributed collaborative infrastructure, would include: distributed data-sharing Simplified data discovery of climate data Large-scale climate data processing and analysis Increased collaboration among climate research scientists Aid in climate assessments and estimates of future climate variability

and trends

For more information on ESG, visit our website at: http://www.earthsystemgrid.org