H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 1 Data Discovery and Basic Processing within the German Collaborative Climate Community Data and Processing Grid (C3Grid) Project Heinrich Widmann and Stephan Kindermann Model and Data / DKRZ / Max-Planck-Institute for Meteorology Hamburg, Germany GO-ESSP at LLNL Livermore, June 19th – 21st, 2006 C3Grid Home: www.c3grid.de
Data Discovery and Basic Processing within the German Collaborative Climate Community Data and Processing Grid (C3Grid) Project. Heinrich Widmann and Stephan Kindermann Model and Data / DKRZ / Max-Planck-Institute for Meteorology Hamburg, Germany. GO-ESSP at LLNL - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 1
Data Discovery and Basic Processing within the German
Collaborative Climate Community Data and Processing Grid (C3Grid)
Project
Heinrich Widmann and Stephan KindermannModel and Data / DKRZ / Max-Planck-Institute for Meteorology
Hamburg, Germany
GO-ESSP at LLNLLivermore, June 19th – 21st, 2006
C3Grid Home: www.c3grid.de
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 2
Overview
• C3Grid Background• Data Analysis Workflows• C3Grid Architecture and Interfaces• Data Discovery and Metadata in C3-
Grid• Data Information Service with
Lucene• Data Access and Preprocessing• Summary
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 3
C3Grid Background
• C3Grid– Status : month 10 of 36 (phase 1)– is the earth system science community grid
within the German D-Grid initiative– D-Grid includes five further community grid
projects (AstroGrid, HEP-Grid, InGrid, MediGrid, TextGrid)– is a community driven grid
Goal is to develop a grid infrastructure appropriate for typical climate analysis workflows
Stepwise introduction and integration
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 4
Requirements
• Metadata• Discovery• Data access(+
preprocessing)
• Security• Scheduling• Complex
processing
Grid technologies
ISO19115 / ISO19139 OAI-PMH + Lucenecommunity
webservice
Shibboleth Globus Toolkit 4 WS-GRAM
C3Grid Data Analysis Workflow Requirements
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 5
C3Grid Architecture and Interfaces
Data
Discovery
Data Access and
Basic Processing
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 6
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 9
C3Grid Portal – Simple search
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 10
C3Grid Portal – Advanced search
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 11
C3Grid Data Access and Preprocessing
• Data access interface– Community-specific webservice (WSDL)– Solutions of the individual institutes will
be adapted to support the webservice•e.g. triggering of local data
processing tools – Support data base and file based
storage types– More detailed use metadata will be
provided during the extraction process with the data
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 12
C3Grid Data Access/Preprocessing Interface
datadata
DB
Files
DataAccessWeb
service
Access
CDO processing
Stage file webservice request contains :• ObjectList of OIDs requested• CFList of standard names • Space constraints• Time constraints• Target directory• File format, e.g. netCDF or grib• …
SOAP-XMLStageFileRequest
Constraints
necessaryprocessing
CF standardnames
Local variable
names
data
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 13
Summary
• Grid development is application driven• Discovery is based on
– ISO 19115/19139 based metadata catalog– Hierarchical, two-leveled metadata
scheme– Text based search in the catalog
• Data access is implemented by• Proprietary C3Grid data access interface
(webservice)
• Part of the use data are provided along with the data extraction
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 14
The end
H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19th 2006 / 15