1 Tracking Metadata and Lineage of the Data Processing Chain for Mapping Snow Cover Properties with the NASA MODIS James Frew 1 , Thomas H. Painter 2 , Peter Slaughter 1 , Jeff Dozier 1 1 Donald Bren School of Environmental Science and Management, University of California, Santa Barbara 2 National Snow and Ice Data Center, University of Colorado, Boulder
33
Embed
1 Tracking Metadata and Lineage of the Data Processing Chain for Mapping Snow Cover Properties with the NASA MODIS James Frew 1, Thomas H. Painter 2, Peter.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Tracking Metadata and Lineageof the Data Processing Chain
for Mapping Snow Cover Propertieswith the NASA MODIS
James Frew1, Thomas H. Painter2,Peter Slaughter1, Jeff Dozier1
1Donald Bren School of Environmental Science and Management, University of California, Santa Barbara2National Snow and Ice Data Center,University of Colorado, Boulder
2
Outline
Motivation Snow mapping product Implications for hydrologic modeling
Lineage Capture Wrapping: the ESSW experience Instrumenting,
overriding,monitoring: the (ongoing) ES3 experience
3
MODIS image – Sierra Nevada
EOS Terra MODIS
07 March 2004
MOD09 Surface Reflectance
0.555 0.645 0.858
44
Snow-covered area and grain size
5
Hindu Kush
2003 DOY 070
6
Colorado RockiesCLPX
13 March 2002
7
Model structure: MODIS snow-area / albedo
Basinmask
Processing Lineage
Watershedinfo
MODIScloud mask
(48 bits)
MODIS 7 land bands (112 bits)
MODIS quality flags
Topography
MODIS snow cover and grain
size
MODISview
angles
Solarzenith,
azimuth
Snowfraction
albedoRMSerror
Vegfraction
Soilfraction
Shadefraction
Open water
fraction
Quality flag
8
Lineage Capture, Take 1
The ESSW experience
9
Using Existing Science Applications
No “standard”Earth science computing environment commercial packages (ArcInfo, MATLAB, …) public packages/models (MM5, MODTRAN, …) locally-developed codes arbitrary combinations of
Example: SST from AVHRR commercial, standalone programs parameters highly customized for UCSB
How do we get these programs to communicate cooperate
with ESSW, without rewriting them?
Navigate(Manual/Automatic)
Receive
Ingest and Calibrate
Rectify
Sea Surface Temp (SST)
SSTMaps
10
Lineage: Current Best Practice
11
Earth System Science Workbench (ESSW)
Producer and consumer issues can both be addressedby a laboratory metaphor
Experiment Network of models … ingesting / synthesizing data … generating products
; clean up {under,over}flow of MODSCAG run;; Input: prefix = prefix for all of the MODSCAG output filenames; ns = number of samples; nl = number of lines; Output: rewrite of the MODSCAG files;; t.h.painter / 1.19.2005
; open snow fileES3_openr,1,string(prefix,'snow.pic')snow=fltarr(ns,nl)readu,1,snow
Probulator reports not universally unique Q: How hook separate reports together? A: Logger assigns UUIDs to
– Data streams
– Processes
– Jobs (workflows)
Lineage not explicit Q: How publish lineage? A: ES3 Core builds serialized graph
32
Thanks to:
Current Mike Colee Stephane Maritorena Dominic Metzger Karl Rittger Dave Siegel
Former Anurag Acharya Rajendra Bose Scott Denning Debbie Donahue Jim Duff Calin Duma Erik Fields Jim Gray Steve Miley Jordan Morris Mark Pelletier Pete Peterson Walter Rosenthal Klaus Schauser Håvar Valeur
33
To Probulate Further… http://www.snow.ucsb.edu : Publications
Bose, R. and Frew, J., 2005. Lineage retrieval for scientific data processing: a survey. ACM Computing Surveys, vol. 37, no. 1, pp. 1-28. doi:10.1145/1057977.1057978
Dozier, J., and Painter, T.H., 2004. Multispectral and hyperspectral remote sensing of alpine snow properties. Annual Review of Earth and Planetary Sciences, vol. 32, pp. 465-494. doi:10.1146/annurev.earth.32.101802.120404
Molotch, N.P., Painter, T.H., Bales, R.C., and Dozier, J., 2004. Incorporating remotely sensed snow albedo into spatially distributed snowmelt modeling. Geophysical Research Letters, 31, L03501 doi:10.1029/2003GL019063
Frew, J. and Bose, R., 2001. Earth System Science Workbench: a data management infrastructure for Earth science products. In: Kerschberg, L. and Kafatos, M. (eds.) 2001. Proceedings, 13th International Conference on Scientific and Statistical Database Management (SSDBM 2001), pp. 180-189. doi:10.1109/SSDM.2001.938550