6e-1 Science Data Products Daryl Swade DMS Systems Engineer S&OC System Design Review #1
Feb 06, 2016
6e-1
Science Data Products
Daryl Swade
DMS Systems Engineer
S&OC System Design Review #1
6e-2
Intro/Overview
Science Data Product design shows what the DMS software should produce. Software design then shows how to generate data products.
JWST science data products generated by DMS are dynamic due to updates to calibration reference data and algorithms, increased knowledge of the science instruments learned in the
early stages of the mission, and DMS software error corrections.
Intent of DMS Science Data Product Generation is to provide data users with basic reduced data products they need to begin detailed analysis.
DMS data products conform to Flexible Image Transport System (FITS) convention. standard archival format for astronomical data
S&OC System Design Review #1
6e-3
DMS Data Flow – Data Product Generation
S&OC System Design Review #1
6e-4
JWST Up-the-Ramp Readout
S&OC System Design Review #1
6e-5
Data Level Definition
S&OC System Design Review #1
Science Data Archive
Object Catalog
Level 0: Science telemetry packet files
Level 1a: Original FITS files
Level 1b: Uncalibrated FITS files
Level 2a: Ramp calibrated exposure FITS files
Level 2b: Fully calibrated exposure FITS files
Level 4: Products of data analysisExtractedobjects
Proposer
Level 3: Calibrated, combined FITS files
Level 0.5: Pod files with packets sorted by exposure/detector
JWST
6e-6
Science Data Product Inputs
S&OC System Design Review #1
DataManagementSubsystem
(DMS)
ProposalPlanning
Subsystem (PPS)
Flight Operations Subsystem
(FOS)
Project Reference Database
Subsystem (PRDS)
Recorded Science Data Files
Observatory Status Files
Calibrated Engineering Data Products
Science Instrument Aperture Files
Datamode Table
Detector Table
Leap Seconds Table
S&OC PRD Version Name
Proposal Information
Planning Information
NIRSpec MSA Pre-images
Observatory Ephemeris
6e-7
Generation of Original Exposure Data Files
S&OC System Design Review #1
science telemetry processing
Level 0 data file
Read science image data
Observatory Status file Data partitioning
o sort data into files by detector and exposure
Reformat data to FITS
Standard FITS keyword information
o generate standard FITS keywords
o include keyword information from science telemetry headers
Level 1a FITS file
o read science image data from science telemetry files
Level 0.5 pod file
6e-8
Level 1 FITS File Structure
S&OC System Design Review #1
Science Extension
Extension Header
Extension DataMIRI Reference Output Extension
Extension Header
Extension DataGroup Information Extension
Extension Header
Extension Data
Primary Header
6e-9
Science Data Cube
single exposure, single detector
S&OC System Design Review #1
6e-10
Image Orientation
Detector and science image frame orientations as seen looking through the optics toward the sky
S&OC System Design Review #1
red – detector frameblue – science framegreen – apertures
NIRCam
NIRSpec
MIRI
NIRISSFGS1 FGS2
Transformation requires rotation, parity flip, and/or shift.
6e-11
Information Flow to Level 1 FITS Headers
S&OC System Design Review #1
Recorded Science Data
File
Observatory Status File
PPS Database
Detector Table
Datamode Table
Science Instrument Aperture File
Level 1a FITS Header
Level 1b FITS Header
Observatory Ephemeris
FOS PPS PRDS DMS
Engineering Database
Calibrated Engineering Data
6e-12
Level 1a-Level 1b Header Keyword Population
S&OC System Design Review #1
engineering
time
o wait for engineering data from the FOS
o incorporate engineering data into science data headers
Incorporate engineering data
Perform Barycentric time
correction
o convert UTC to barycentric time for a reference pixel of each SCA
Perform Heliocentric time
correctiono convert UTC to heliocentric time
for a reference pixel of each SCA
proposal, planning, and scheduling
Populate PPS infoo populate keyword values
with proposal, planning, and scheduling information
Verify telemetry
o compare information from the science telemetry packet header and engineering parameters to PPS database entries
PPS replicated database tables
position
o populate header keywords with JWST position and velocity information
Determine s/c position and
velocity
Calculate WCS parameters
o translate focal plane coordinates to sky coordinates
o determine differential velocity aberration correction
Level 1b FITS file
Calibrated engineering data
Level 1a FITS file
s/c ephemeris
data
Science Instrument
Aperture File
Observatory Status file Populate OSF info
o populate keyword values with Observatory Status File information
6e-13
Files in an Exposure Data Set
File type Data level created
Science Instrument
Format
uncalibrated 1b all FITS pixel data image and group information ASCII table
MSA metadata 1b NIRSpec FITS ASCII table
reference stars 1b NIRSpec FITS ASCII table
jitter file 1b all FITS ASCII table
count rate image 2a all FITS image
calibrated image 2b all FITS image
processing history 1a-2b all text
S&OC System Design Review #1
6e-14
Exposure File Naming Convention
At data levels 1 and 2, the science data files are FITS format files that contain the pixel values
for a single exposure from a single detector.
Goals for file name: DMS data user able to map the data file to the observation input through APT. Operations staff can map every exposure to a construct in the JWST commanding scripts.
The file name is based on the following exposure hierarchy: Program (PPPPP) Observation number (OOO) Visit (VVV) Visit Group (GG) Parallel Sequence id (S, 1 prime, 2-5 parallel) Activity (AA, base 36) Exposure number (EEEEE)
Example: jw<PPPPP><OOO><VVV>_<GGSAA>_<EEEEE>_<detector >_<suffix> .fits jw12345012009_01101_00012_nrcalong_uncal.fits
Visit ID - <PPPPP><OOO><VVV> - explicitly embedded in exposure file name
Exposure number resets with each new activity The exposure number within the file name does not necessarily monotonically increase for each
exposure within a single visit.S&OC System Design Review #1
6e-15
Guide Star Pipeline
Guide Star Pipeline detailed design to be addressed in DMS SDR4
Jitter data file generated by Guide Star Pipeline Separate process from science data pipelines Data product created at visit level
Guide Star (GS) acquisition images About GS - Full frame image, Acq 1: 128x128, Acq 2: 32x32 At science attitude: Acq 1: 128x128, Acq 2: 32x32
Fine guide box: 8x8 @ 16Hz (64 msec.) May not want to save all 8x8 fine guide data Centroids in engineering data for jitter
Include additional pointing information in engineering data
S&OC System Design Review #1
6e-16
NIRSpec MSA Data Flow
S&OC System Design Review #1
Data Processing
Proposal Input
MSA Pre-
image
MSA Planning
Tool
PPS Database
APT
Science Data
Processing
Recorded Science Data File
Level 1b Science Data File
Level 2 Calibration
Level 2b Science Data File
Level 3 Calibration
Level 3 Science Data File
MSA Metadata
File
JWSTDetector/
exposure based
Source based
PI
Source Catalog
Data product key:Ancillary dataExposure dataCombined data
6e-17
NIRSpec MSA Metadata File Structure
S&OC System Design Review #1
Data array for each shutter planned open or closed
Slitlet information, background shutter, fraction of source in shutter
Source id, source name, alias, RA, Dec
6e-18
Generation of Calibrated Exposure Data Files
S&OC System Design Review #1
Level 1b FITS file
calibration reference files
calibration
Perform up-the-ramp calibration
o remove detector specific signatures
Determine calibration
reference files
o determine appropriate calibration reference files and switches
Perform instrument specific calibration
Calibration reference
files
Level 2a data file
Level 2b FITS file
CRDS
6e-19
Level 2 FITS File Format
S&OC System Design Review #1
Primary Header
Data Quality Extension
Extension Header
Extension Data
Science Extension
Extension Header
Extension Data
Error Extension
Extension Header
Extension Data
Additional Extension(s)
Extension Header
Extension Data
MIRI Reference Output Extension
Extension Header
Extension Data
6e-20
Level 3 Data Products - Associations
Relationships between multiple exposures are captured in an association.
An association is a means of identifying a set of exposures that belong together and may be dependent upon one another.
The association concept permits exposures to be calibrated, archived, retrieved, and reprocessed as a set rather than as individual objects.
For initial processing, an association and its level 3 data products will not extend beyond a single observation. Design will include the capability to support multi-observation and
multi-program associations.
For each association, DMS will generate the most combined and least combined data products. Intermediate data products will not be saved. For imaging associations, the most combined will be either a
mosaic or a dither product.
S&OC System Design Review #1
6e-21
Association Components
S&OC System Design Review #1
Science Instrument
Data Products Related Files Contemporaneous Calibration Files
NIRCam • Mosaic combined image (multiple detectors)
• Dither combined image (if no mosaic)
• Individual exposures (single detector; levels 1a, 1b, 2a, 2b)
• Coronagraphic target acquisition
• Jitter data
• Flats (taken with science exposure)
• Persistence darks
NIRSpec • Extracted 1-d and 2-d spectra over both detectors
• MSA Source Catalog (level 4)
• Individual exposures (single detector; levels 1a, 1b, 2a, 2b)
• Target acquisition images
• Confirmation images
• Jitter data
• MSA Pre-image
• MSA Metadata file
• Reference star file
• Background exposures (may be shared among exposures)
• Auto-flats
• Wave-cals
MIRI • Mosaic combined image
• Dither combined image (if no mosaic)
• Extracted 1-d and 2-d spectra
• Extracted 1-d and 2-d spectra over both IFU detectors
• Individual exposures (single detector; levels 1a, 1b, 2a, 2b)
• Coronagraphic target acquisition images
• Spectroscopic target acquisition images
• Jitter data
• Imager internal flats
• MRS internal flats
• Chopped background image (different visit, no interrupts)
• Persistence darks
NIRISS/FGS • Mosaic combined image
• Dither combined image (if no mosaic)
• Individual exposures (single detector; levels 1a, 1b, 2a, 2b)
• Jitter data
• Point source image for PSF characterization
• Persistence darks
6e-22
Association Data Products
Association Pools Created by Science Data Processing Contains a list of all potential exposures that may form an
association
Association Table Created in calibration pipeline environment from Association Pool
information Contains list of exposures to be used to create a single Level 3
Data Product
Level 3 Data Products Formed from exposures listed in Association Table
S&OC System Design Review #1
6e-23
Association Processing
All exposures – even single, unassociated exposures – will have a level 3 data product. Well-established image rectification software (Drizzle) will be run at
level 3 on all exposures. All data will get a source based, level 3 file name.
S&OC System Design Review #1
Assemble exposures
from level-0 data
Level-1 processing
Association pool
Uncaibrated exposure
Level-2a processing
Partially calibrated exposure
Level-2b &Level-3
processing / Association Generator
Associated data
products
Hold for all observation exposures
6e-24
Primary Header
Science Extension
Extension Header
Extension DataHeader Table Extension
Extension Header
Extension Data Additional Extension(s)
Extension Header
Extension Data
Level 3 Image FITS File Format
S&OC System Design Review #1
Most combined image
FITS table of selected keywords from input exposures
Additional extensions to be defined by calibration working group
6e-25
Level 3 Data Product File Name
Shift in naming convention for level 3 data product file names
The association data product name will provide an indication of the file content. Based on data discovery lessons learned from MAST High Level Science Product
(HLSP) and Hubble Legacy Archive (HLA) experience, A proposed name for the JWST level 3 data products was patterned after the
HLSP data file names. (http://archive.stsci.edu/hlsp/hlsp_guidelines.html#Filenames).
The format for the level 3 data product file name would be:
jw<ppppp>_<target(epoch)>_<science_instrument>_
<optical_element>_ <product_name>.fits <ppppp> is the program identifier <target> is the target name, which may contain proposer defined non-
alphanumeric characters and an epoch <science_instrument> is nircam, nirspec, miri, niriss, or fgs < optical_element > is the optical element or optical element combination used <product_name> indicates the type of data product, e.g., image
Examples from SODRM: jw94010_m31_nircam_f356w_image.fits jw95020_cdfs-07-e1_nircam_f115w_image.fits jw93060_w51a+g49.5-0.4_miri_f1500w_image.fits
S&OC System Design Review #1
6e-26
Data Level Format Summary
Data level Format
Level 0 CCSDS packets
Level 0.5 CCSDS packets sorted by exposure and detector
Level 1a 4-d data cube
Level 1b 4-d data cube
Level 2a 3-d data cube – collapse groups within each integration into a count-rate imageandSingle 2-d image – collapse integrations into one count-rate image
Level 2b Single 2-d image
Level 3 Single or combined, rectified 2-d image; extracted spectra
S&OC System Design Review #1