A Data Quality Screening Service for Remote Sensing Data Christopher Lynnes, NASA/GSFC (P.I.) Edward Olsen, NASA/JPL Peter Fox, RPI Bruce Vollmer, NASA/GSFC Robert Wolfe, NASA/GSFC Contributions from R. Strub, T. Hearty, Y-I Won, M. Hegde, S. Zednik, P. West, N. Most, S. Ahmad, C. Praderas, K. Horrocks, I. Tchered, and A. Rezaiyan-Nojani Advancing Collaborative Connections for Earth System Science (ACCESS) Program Project Page: http://tw.rpi.edu/web/project/DQSS
39
Embed
A Data Quality Screening Service for Remote Sensing Data Christopher Lynnes, NASA/GSFC (P.I.) Edward Olsen, NASA/JPL Peter Fox, RPI Bruce Vollmer, NASA/GSFC.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
A Data Quality Screening Service for Remote Sensing
Data
Christopher Lynnes, NASA/GSFC (P.I.)Edward Olsen, NASA/JPL
Peter Fox, RPIBruce Vollmer, NASA/GSFCRobert Wolfe, NASA/GSFC
Contributions fromR. Strub, T. Hearty, Y-I Won, M. Hegde, S. Zednik, P. West,
N. Most, S. Ahmad, C. Praderas, K. Horrocks, I. Tchered, and A. Rezaiyan-Nojani
Advancing Collaborative Connections for Earth System Science (ACCESS) Program
Surface Type Flag0=Ocean, deep lake/river1=Coast, shallow lake/river2=Desert3=Land
Big-endian arrangement for the Cloud_Mask_SDS variable in atmospheric products from Moderate Resolution Imaging
Spectroradiometer (MODIS)
8
Current user scenarios...
Nominal scenario Search for and download data Locate documentation on handling quality Read & understand documentation on quality Write custom routine to filter out bad pixels
Equally likely scenario (especially in user communitiesnot familiar with satellite data) Search for and download data Assume that quality has a negligible effect
Repeat for
each user
9The effect of bad qualitydata is often not
negligible
Total Column Precipitable Water Quality
Best Good Do Not Usekg/m2
Hurricane Ike, 9/10/2008
10Neglecting quality may introduce bias (a more subtle
effect)AIRS Relative Humidity Comparison against Dropsonde with and without Applying PBest Quality Flag Filtering
Boxed data points indicate AIRS RH data with dry bias > 20%
From a study by Sun Wong (JPL) on specific humidity in the Atlantic Main Development Region for Tropical
Storms
11Percent of Biased Data in MODIS Aerosols Over Land Increases as Confidence Flag
Decreases
Bad
Marginal
Good
Very Good
0% 20% 40% 60% 80% 100%
Compliant*Biased LowBiased High
*Compliant data are within + 0.05 + 0.2τAeronet
Statistics derived from Hyer, E., J. Reid, and J. Zhang, 2010, An over-land aerosol optical depth data set for data assimilation by filtering, correction, and aggregation of MODIS Collection 5 optical depth retrievals, Atmos. Meas. Tech. Discuss., 3, 4091–4167.
12
Making Quality Information Easier to Use
via the Data Quality Screening Service (DQSS)
.
13
DQSS Team
P.I.: Christopher Lynnes
Software Implementation: Goddard Earth Sciences Data and Information Services Center Implementation: Richard Strub Local Domain Experts (AIRS): Thomas Hearty and Bruce
Vollmer
AIRS Domain Expert: Edward Olsen, AIRS/JPL
MODIS Implementation Implementation: Neal Most, Ali Rezaiyan, Cid Praderas,
Karen Horrocks, Ivan Tchered Domain Experts: Robert Wolfe and Suraiya Ahmad
Semantic Engineering: Tetherless World Constellation @ RPI Peter Fox, Stephan Zednik, Patrick West
14The DQSS filters out bad pixels for the user
Default user scenario Search for data Select science team recommendation for quality
screening (filtering) Download screened data
More advanced scenario Search for data Select custom quality screening parameters Download screened data
15
DQSS replaces bad-quality pixels with fill values
Mask based on user criteria(Quality level
< 2)
Good quality data pixels
retained
Output file has the same format and structure as the input file (except for extra mask and original_data fields)
Original data array(Total column precipitable water)
16Visualizations help users see the effect of different quality filters
17DQSS can encode the science team recommendations on
quality screening
AIRS Level 2 Standard Product Use only Best for data assimilation uses Use Best+Good for climatic studies
MODIS Aerosols Use only VeryGood (highest value) over
land Use Marginal+Good+VeryGood over
ocean
18
Initial settings are based on Science Team recommendation.
(Note: “Good” retains retrievals that Good or better).
You can choose settings for all parameters at once...
...or variable by variable
Or, users can select their own criteria...
19
DEMO
http://mirador.gsfc.nasa.gov (Search for ‘AIRX2RET’)
20
DQSS Under the Hood
21
DQSS Flow
user selection
ancillary file
Screener
Quality Ontology
data file
quality mask
screened data file
EndUser
data file w/ mask
Masker
Ontology Query
22DQSS Ontology(The Whole Enchilada)
23
DQSS Ontology (Zoom)
24
DQSS Status
25Significant Accomplishments
Release 1.0 of DQSS for AIRS Level 2 Standard Retrieval Technology Readiness Level (TRL) = 9 (for AIRS L2) Includes Quality Impact views Disclosure of Invention (NF 1679) filed Announced to AIRS Registered Data Product User Community
(~770)
Papers and Presentations Managing Data Quality for Collaborative Science workshop (peer-
reviewed paper) Sounder Science meeting ESDSWG Poster A-Train User Workshop (part of GES DISC presentation) AGU: Ambiguity of Data Quality in Remote Sensing Data
Ontology (v. 2.4) to accommodate both AIRS and MODIS L2
26
Current Status
DQSS is operational for AIRS L2 Standard Products DQSS is offered through the Mirador data
search interface* at the GES DISC
DQSS has been refactored to work in LAADS/MODAPS environment 21/2 month delay due to refactoring and
(successful) audit of NPR 7150.2 compliance by NASA’s Office of the Chief Engineer
Schedule no longer has room for client-side screening
But maybe...(to be continued)* http://mirador.gsfc.nasa.gov