Presented at the GHRC User Working Group Meeting October 7, 2015 GHRC DATA PROCESSES Lifecycle, Levels of Service, Maturity Model Helen Conover GHRC Operations Manager [email protected]
Presented at the GHRC User Working Group Meeting October 7, 2015
GHRC DATA PROCESSES Lifecycle, Levels of Service, Maturity Model Helen Conover GHRC Operations Manager [email protected]
GHRC Dataset Lifecycle Formalized GHRC dataset management processes in Lifecycle and Levels of Service documents • Reviewed lifecycle documents from NOAA and multiple
DAACs (NSIDC, PO.DAAC, LP DAAC)
• Reviewed GHRC practices and procedures
• Assessed GHRC on Peng’s stewardship maturity matrix for digital environmental data
https://ghrc.nsstc.nasa.gov/home/ghrc-docs/data-management
10/7/2015 2 User Working Group Meeting
Peng, G., Privette, J. L., Kearns, E. J., Ritchey, N. A., & Ansari, S.. (2015). A Unified Framework for Measuring Stewardship Practices Applied to Digital Environmental Datasets. Data Science Journal, 13(0), 231–253. DOI: http://doi.org/10.2481/dsj.14-049
New Dataset Evaluation
10/7/2015 3 User Working Group Meeting
DAAC Process for Implementing New Data Types and/or Services (As Is)
3.0
ES
DIS
Pro
ject
2.0
DA
AC
1.0
DA
AC
Use
r Wor
king
G
roup
4.0
NA
SA
HQ
Ear
th S
cien
ce D
ata
Sys
tem
E
xecu
tive
Identify request for supporting
new data type or service
2.2ESDIS Review
Required
Implement New Data Type or
Service
Review Holdings, Product
Templates, and Impact
Assessments
Complete Product Templates and
Impact Assessments
Review New Request
3.1NASA HQ
review required?
Generate Rejection
Justification
Review/Update Rejection
Justification
3.2Approve Request?
Review New Request
4.0Approve
Request?
Review/Update Rejection
Justification
1.2Modify
Request?
Return to Start
End
StartYes
No
Yes
No
Yes
No
No
Yes
Yes
No
2.1UWG review
required?
1.1Recommend
Implementation?
Yes
No
Archival Interest Form
DAAC appropriate
?
Email DP with appropriate alternate archives
DP
DC
Data Provider
Dataset Coordinator
DP
DC
Dataset Ingest Process Planning Ingest Documenta<on Publica<on
Answer Data
Provider Ques<ons
Upload Sample data
Confirm Submission
Collect ini<al metadata
Assign soBware developer
Verify Data Set
completeness
Publish Data Set
Monitor submission
Ini<ate data set
submission
Send ini<al email to DP
Verify data file names and loca<ons
Assign Documenta<on Coordinator
Create/Edit Metadata
Review landing page and guide doc
Provide documents
Ingest / archive
dataset and documents
Configure ingest/archive
soBware
Rename / reformat scripts (if needed)
Data Provider
Dataset Coordinator
SoBware Developer
Documenta<on Lead
DP
DC
SW
DL
Outreach OR
News items: • Weekly notes • Social media • GHRC web site
Op<onal: • Earthdata feature • Email announcement
DP
DC
SW
DL
OR
DP
DC
SW
DL
OR
Thanks to ORNL DAAC to swimlanes graphic
New Dataset Versions Planning Ingest Documenta<on Publica<on
No<fy DAAC that
new version is available
Confirm Submission
Update ini<al metadata
Assign soBware developer
Verify Data Set
completeness
Publish Data Set
Configure ingest/archive
soBware
Verify data file names and loca<ons
Assign Documenta<on Coordinator
Provide updated
documents
Ingest / archive
dataset and documents
Review / update rename /
reformat scripts (if needed)
Answer Data
Provider Ques<ons for new version
Create/Edit new version Metadata Review both
landing pages and guide
docs Update previous
version metadata to reference new
version
Con<nue to Re(re Dataset
Re<re previous version?
News items: • Weekly notes • Social media • GHRC web site
Op<onal: • Earthdata feature • Email announcement
DP
DC
SW
DL
OR
DP
DC
SW
DL
OR
Data Provider
Dataset Coordinator
SoBware Developer
Documenta<on Lead
DP
DC
SW
DL
Outreach OR
Retire a Dataset
Op<ons to re<re a dataset: ① Leave data available online with low level of service ② Remove data from online server and public catalog,
keep on archive ③ Remove from online server, catalog and archive ④ Transi<on to long term archive
Request to re<re a dataset
Prepare Data Assessment Package
Re<re?
Request to re<re a dataset
Remove Data Set
Assign Documenta<on Coordinator
Sta<c landing page, reference new version if applicable
No more metadata or
service updates
Re<re Request Evalua<on Documenta<on Re<re Dataset
Package data, metadata and
docs
1
Transi<on Data Set
DP
DC
UWG
DL
ESDIS Re<re?
4 2
3
NASA ESDIS Project
Data Provider
Dataset Coordinator
DP
DC
User Working Group UWG
Documenta<on Lead DL
ESDIS
DP
DC
UWG
DL
ESDIS
1 2 3 4
DAAC Process for Implementing New Data Types and/or Services (As Is)
3.0
ESD
IS P
roje
ct2.
0D
AAC
1.0
DAA
C U
ser W
orki
ng
Gro
up
4.0
NAS
A H
QEa
rth S
cien
ce D
ata
Syst
em
Exec
utiv
e
Identify request for supporting
new data type or service
2.2ESDIS Review
Required
Implement New Data Type or
Service
Review Holdings, Product
Templates, and Impact
Assessments
Complete Product Templates and
Impact Assessments
Review New Request
3.1NASA HQ
review required?
Generate Rejection
Justification
Review/Update Rejection
Justification
3.2Approve Request?
Review New Request
4.0Approve
Request?
Review/Update Rejection
Justification
1.2Modify
Request?
Return to Start
End
StartYes
No
Yes
No
Yes
No
No
Yes
Yes
No
2.1UWG review
required?
1.1Recommend
Implementation?
Yes
No
ESDIS-‐UWG review process
Levels of Service
Data collections at the GHRC DAAC may be handled with different levels of service (LoS). • For some aspects of data services, such as ingest
method, LoS corresponds to characteristics of the data. • For other aspects of data services, LoS will depend on
overall data handling priority assigned to the general categories of GHRC data holdings
CATEGORIES*OF*DATA*SERVICES*
Off/site*Backup* Data*Ingest*Post/Ingest*Processing*
Metadata*and*Documentation*
Distribution*Services*
Cloud,'other'DAAC'
Automated,'ongoing'
Product'generation' Guide'document' Exploration,'
analytics'Tape'copy' Periodic'ingest' Reformat' README' Visualization'PI'institution' Bulk'download' Rename' DOI'and'citation' Access'services'' PI'upload' None' Catalog' FTP/HTTPS'
Dataset Priorities Priority' '''''''''''''''''''''''''''''''''GHRC'DATA'CATEGORIES''
SATELLITE'MISSIONS'1" NASA"satellite"datasets"(OTD,"TRMM"LIS,"ISS"LIS,"AMSU)"1" Airborne"validation"datasets"(LIP,"multiple"campaigns)"2" Ground"validation"datasets"–"open"access"(LMA)"3" Other"satellite"datasets"(DMSP"OLS,"NOAA"MSU)"5" Ground"validation"datasets"–"commercial,"restricted"access"
(Vaisala/NLDN,"WWLLN,"ENGLN)"MEaSUREs'PROGRAM'
1" DISCOVER"(RSS)"FIELD'CAMPAIGNS'and'EARTH'VENTURES'(Hurricane'Science'or'GPMAGV)'1" NASA"research"instruments"(airborne"or"ground,"NASANsponsored"PI)"2" Affiliated"research"instruments"(e.g.,"from"partner"university)"3" Other"agency"research"instruments"(e.g.,"sponsored"by"NOAA,"DOE)"4" Ancillary"research"data"(e.g.,"PERSIANN,"TRMM"flood"maps)"5" Other"agency"operational"data"(e.g.,"GOES"imagery,"NWS"radar)"
NASA'APPLICATIONS'Research'Results'1" Applications"products"(e.g.,"SANDS"analysis"products)"3" Selected"input"products"(e.g.,"MODIS"subsets"for"selected"storms)"
Data Maturity Model Recommendation 10: Develop a data maturity model for GHRC data. Provide this on website and include maturity information for each dataset provided. Review NOAA’s data maturity model as a starting point. • Also looked at NASA’s data maturity levels
o Beta – gain familiarity with data parameters and formats o Provisional – initial data exploration and process studies o Validated Stage 1 – selected independent measurements o Validated Stage 2 – peer reviewed literature o Validated Stage 3 – quantified uncertainty o Validated Stage 4 – systematic validation updates
10/7/2015 9 User Working Group Meeting
NOAA: http://www1.ncdc.noaa.gov/pub/data/sds/maturity-table-6level.pdf NASA: http://science.nasa.gov/earth-science/earth-science-data/data-maturity-levels/
THANK YOU for your attention Questions? Please contact GHRC User Services for any help or questions [email protected]
10/7/2015 10 User Working Group Meeting