PDS4 Build 3b Status and Readiness Steve Hughes MC Face-to-Face Columbia, Maryland April 2-3, 2013
Feb 24, 2016
PDS4 Build 3bStatus and Readiness
Steve Hughes
MC Face-to-FaceColumbia, Maryland
April 2-3, 2013
2
Topics
• Build 3b Deliverables• Candidate Classes for V1.0• Documents
• Overview of the Information Architecture• Summary of Progress to Date• Testing• Future Plans• Summary• Acknowledgements
3
Candidate Classes For V1.0Fundamental Data Structures1
Class Subclasses DescriptionArray Array_2D,
Array_2D_ImageArray_2D_SpectrumArray_3DArray_3D_Spectrum
The Array class defines a homogeneous N-dimensional array of scalars.
Table (Base) Fixed-width Character Table, Fixed-width Binary Table
The Table class defines a set of repeating records, each with a heterogeneous set of fields.
Parsable Byte Stream
Delimited Table, Header, SPICE Kernel, Stream Text
The Parsable Byte Stream class defines byte streams that have standard parsing rules.
Encoded Byte Stream
Encoded_Image, PDF/A The Encoded Byte Stream class defines byte streams that must be decoded by software before use. These byte streams must only use standard encodings.
1And all component and supporting classes.
4
Base Products1
and Aggregate ProductsClass Subclasses DescriptionProduct Observational N/A A Product_Observational is a set of one or more
information objects produced by an observing system.
Product Document N/A The Product Document class describes a document.
Product XML_Schema N/A The Product_XML_Schema is used to describe XML schemas.
1And all component and supporting classes.
Class Subclasses Description
Product Collection N/A A Product_Collection has a table of references to one or more data products.
Product Bundle N/A A Product_Bundle has a table of references to one or more collections.
Documents1
5
• Information Model Specification – Version 0.3.1.0.b• XML Schemas – Version 0.3.1.0.b• Data Dictionary – Version 0.3.1.0.b• Concepts Document – Version 0.4.0• Standards Reference - Version 4.0.8• Data Providers Handbook - Version 0.3.10• PDS4 Example Products - Version 0.3.1.0.b
1Posted to http://pds.nasa.gov/pds4
Products for Internal Use1
6
• PDS4 Context Products• Instrument, Investigation, Node, …
• “PDS3” Context Products• For migration and archive of PDS3 catalog
information.
• PDS4 Data Dictionary Schema
1And all component and supporting classes.
Information Architecture Concepts
7
Tagged Data Object (Information Object)
<Array_2D_Image> <local_identifier>MPFL-M-IMP_IMG_GRAYSCALE… <offset unit="byte">0</offset> <axes>2</axes> <axis_index_order>Last Index Fastest… <Element_Array> <data_type>UnsignedMSB2</data_type> <unit>data number</unit> </Element_Array> <Axis_Array> <axis_name>Line</axis_name> <elements>248</elements> <unit>not applicable</unit> <sequence_number>1</sequence_number> </Axis_Array> <Axis_Array> <axis_name>Sample</axis_name> <elements>256</elements> <unit>not applicable</unit> <sequence_number>2</sequence_number> </Axis_Array></Array_2D_Image>
Label Schema
Used to Create
Describes
Extracted/Specialized
InformationModel
Data Object
Data Element
Class
has
Planetary ScienceData Dictionary
Expressed As
Product
Validates
8
Architecture Goals• Setup an information model development and
management framework that includes methodologies and tools to address the following tasks across geographical, political, and domain boundaries. 1.Develop and manage an archive information model
for diverse and complex planetary science domains 2.Maintain the information model independent of
system implementation choices.3.Use the information model to drive implementation
and documentation.4.Enable and maintain interoperability between
domains.5.Manage continuous evolutionary changes and
additions to the information model.
Requirements & Domain Knowledge
PDS4 Information
Model
Query Models
Information Model
Specification
XML Schema(pds)
Filter and Translator
Protégé Ontology Modeling
Tool
PDS4 Data Dictionary
(Doc and DB)
XML Document
(Label Template)
XMI/UML
Registry Configuration Parameters
PDS4 Data Dictionary
(ISO/IEC 11179)
PDS4 Model-DrivenProcess
Ontology contents are extracted, filtered,
translated, and then written to PDS4 system
files.
10
PDS4 Concept Map
11
PDS4 Documentsand their Relationships
ConceptsDocument
Big Picture
StandardsReference
RequirementsUser Friendly
XML Schemas
Blueprints
PDS4Product Labels
Deliverables
Data Dictionary
Definitions
PDS4 InformationModel Specification
RequirementsEngineering Specification
Data DictionaryTutorial
Informative Data Provider’sHandbook
Cookbookde
rive
generate
references
create /validate
inst
ruct
generate
refe
renc
es
RegistryConfiguration File
Object Descriptionsconfiguregenerate
Registry
Product Tracking and Cataloging
gene
rate
InformativeDocument
StandardsDocument
File
System
Legend
12
Referenced Standards• Open Archival Information System (OAIS) Reference Model - Defines the
“Information Object” a key component of the model.
• ISO/IEC 11179-3: Registry Metamodel and Basic Attributes - Provides the schema for the data dictionary. Defines the concepts of registration authority and steward for governance.
• CCSDS 312.0-G-1: Reference Architecture for Space Information Management
• Object_Oriented Data Modeling – Used as the modeling methodology.
• XML/XML Schema – Provides the label syntax and validation mechanism.
• OASIS/ebXML Registry Information Model - Provides attributes for object registration within a federated registry/repository.
• ISO 15836:2009 The Dublin Core Metadata Element Set – Provides standard web resource identification attributes.
• Semantics - RDF, RDFS, OWL - Provides W3C standards for knowledge representation.
13
Summary of Progress to Date(Since Inception)
• Continuously applied the model-driven paradigm• Use an Ontology Modeling Tool to capture and validate the PDS4
Information Model. • Automatically generate the Information Model Specification Document,
Data Dictionary Document, XML Schemas, and other artifacts. • Participated in all Builds: build 1(a,b,c,d), 2 (a,b,c), 3(a)• Participated in all major reviews and tests
• E.g. Beta test conducted by Atmospheres to get user input on PDS4 concepts (XML, Bundles, etc)
• PDS4 Data Standards were used to design and develop data product labels for the LADEE and MAVEN teams
• PDS4 Data Standards are endorsed and in use by IPDA• International implementation by PSA for Bepi Colombo underway• International access prototype demonstrated at IPDA 2012 SC
• PDS4 Data Standards are included in all deployed operational capabilities• Registry and harvest infrastructure at EN• Central catalog migration to registry; High level search• PDS3 data ingestion into PDS4 system at EN
14
Summary of Progress to Date(Since Build 3a, Sept 2012)
• Applied 140 updates to the PDS4 Information Model based on DDWG design decisions and RFA/Issue resolutions.• Summary of Changes
• Attribute and Class names, Cardinality, Enumerated Values and Value Meanings, Schematron Rules and Patterns
• Many new RFAs had been previously addressed or reviewed and rejected.
• Very few structural changes.• Document Alignment Team tightened up the Standards
Reference and brought it into alignment with the Information Model.• Remaining documents have been reviewed and
updated.• Released Version 0.3.1.0.b of the PDS4 Data Standards on
3/28
15
DDWG RFA Status
• 8 RFAs PENDING/DEFERRED involving:• Encoding Type • Target Identification (SBN)• Array Axis Naming (non-Imaging)
• Build 2c• OPEN
0• PENDING
1• DELIVERED
159• CLOSED
108• DEFERRED
2• TOTAL
270
• Build 3a• OPEN
0• PENDING
0• DELIVERED
13• CLOSED
8• DEFERRED
5• TOTAL
33
Testing• Oxygen is used to test the generated XML Schemas
and Schematron files for validity and well-formedness.
• File Difference (Ultra-Edit Compare) is used to identify differences between current and prior versions of generated files after updates.• XML Schema, Schematron, Information Model
Specification, ISO/IEC-11179 Data Dictionary files.• Regression tests are performed against a suite of
example products.• Regression tests are performed against selected
node bundles (Imaging, Atmospheres)
Support for System Testing• Will track RFAs against PDS4 Data Standards
that are produced by PDS Node Bundle development going forward.
• RFAs will be reviewed by DDWG for recommendation and disposition.• RFAs will be classified based on impact to
system software, schema, schematron, etc.
18
Plans• Small teams will continue to be formed for specific
issues • Teams report back to DDWG with recommendations and final vote.
• Current teams• Geometry White Paper – E. Guinness, C. Acton• Cartography – E. Rye, C. Isbell
• DDWG will again scale back to 1 hour a week and meet as needed.
• Finalize V1.1 classes, e.g. Image_2D_Image• Address current Pending and Deferred RFAs• Update and re-institute Standards Change Process for PDS4
19
Task
Title: Geometry White Paper
Team lead and members: E. Guinness, C. Acton
Short Description: Interview geometry experts from the various PDS disciplines and write a white paper that describes the scope and a set of high-level requirements for Geometry.
Goals: Collect a PDS-wide set of requirements for Geometry
Schedule (Major milestones): Dec – Start Gathering Geometry Requirements May – Distribute Draft for Review and Comment Jul – Finalize and Distribute White Paper.
20
TaskTitle: Cartography
Team lead and members: E. Rye, C. Isbell
Short Description: Using existing and accepted cartography standards extend/develop a PDS4 cartography model that can be used across the PDS discipline nodes and also extended as needed for specific purposes.
Goals: Develop a cartography model suitable for use across the PDS discipline nodes.
Schedule (Major milestones): Apr – Continue Development May – Test ingest Cartography Model into PDS4 Sep – Release for general use in next Build.
21
Summary
• The PDS4 Data Standards are currently more stable than the PDS3 Data Standards ever were. 1
• The PDS4 Data Standards have been tested extensively.• The majority of new RFAs and issues are well focused,
relatively minor, and in many cases either already addressed or previously rejected.
• The PDS4 Data Standards are already in use.• Missions: LADEE, MAVEN, Bepi, Phoenix, etc• System: PDS4 Catalog and system dictionary
• The PDS4 System is resilient and will be able to adapt to change.• Information model is independent from the
implementation (XML Language and Technologies and System Software)
1 Quote from IPDA member.
Acknowledgements*
* Anyone who sat through a DDWG 2-hour telecon or provided useful input.
Ed BellRichard ChenDan CrichtonAmy CulverPatty GarciaEd GrayzeckEd GuinnessMitch GordonSean HardmanLyle HuberSteve HughesChris IsbellSteve JoyRonald Joyner
Debra KazdenTodd KingJohn KodisJoe MafiMike MartinThomas MorganLynn NeakrasePaul RamirezAnne RaughShannon ReesMark RoseMatias RoybalElizabeth RyeBoris Semenov
Dick SimpsonSusie SlavneyDillon White
Peter AllanDavid HeatherMichel GangloffSanta MartinezThomas RoatschAlain Sarkissian
23
Questions and Answers
24
Backup
Product_Observational Label
25
Identification_AreaLogical_IdentifierVersion_Id
Observation_AreaTime_CoordinatesPrimary_Result_DescriptionInvestigation_AreaObserving_SystemTarget_Identification
Reference_ListInternal_ReferenceExternal_Reference
File_Area_ObservationalFile
HeaderArray_2d_Image …
CartographyNode_AreaMission_Area
26
• digital object: An object which is real data — for example, a binary image of a redwood tree or an ASCII table of atmospheric composition versus altitude.
• physical object: An object which is physical or tangible (and, therefore, does not itself fit into a digital archive). Examples of ‘physical objects’ include the planet Saturn and the Venus Express magnetometer.
• conceptual object: An object which is intangible (and, because it is intangible, does not fit into a digital archive). Examples of ‘conceptual objects’ include the Cassini mission and NASA’s strategic plan for solar system exploration.
27
Fundamental Data Standards
Core Components in Context
The PDS
ExternalResources
Internet Users
Internet Service Endpoints Search, Retrieve, Transport, Transform, …
Product Centric
DisciplineCurated
Data
29
Plans for Build 3B
• Remaining Topics (To be prioritized)
• Calibration
• Processing History (AMMOS)
Build 3b Deliverables*
30
• Software System• Registry Service• Harvest Tool• Validate Tool • Security Service• Report Service• Search Service • CI Tool • Upgraded portal search and
page views to support PDS4
• Data Standards*• Information Model• XML Schemas• Data Dictionary• Concepts Document• Standards Reference• Data Providers Handbook• PDS4 Example Products
* Posted to http://pds.nasa.gov/pds4