COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 1 of 24 Science Data Processing

COS PIPELINE PDRDaryl Swade

December 7, 2000 OPUS / OTFR

Space Telescope Science Institute

1 of 24

Science Data Processing

www.stsci.edu/software/OPUS/

www.dpt.stsci.edu/




2 of 24

General OPUS Requirements for Science Data Processing OPUS will develop the COS pipeline based on the existing OPUS will develop the COS pipeline based on the existing

pipeline model including OTFR.pipeline model including OTFR. Level 0 data packetized and Reed-Solomon corrected by PACOR Level 0 data packetized and Reed-Solomon corrected by PACOR

at GSFCat GSFC Receive science telemetry (level 1a data) at STScI as science Receive science telemetry (level 1a data) at STScI as science

instrument specific ‘pod files’instrument specific ‘pod files’– Engineering snapshot includedEngineering snapshot included– No on-board compression for COS dataNo on-board compression for COS data

STScI processing on Compaq ALPHA/Tru64 UNIX platformSTScI processing on Compaq ALPHA/Tru64 UNIX platform




3 of 24

General OPUS Requirements for Science Data Processing (cont.) OPUS must account for all scheduled exposures.OPUS must account for all scheduled exposures. Convert telemetry to FITS formatConvert telemetry to FITS format

– Structure tables or data arrayStructure tables or data array– Populate header keywordsPopulate header keywords

> Keywords to provide metadata for archive catalogKeywords to provide metadata for archive catalog

Associate groups of exposures that must process further as a Associate groups of exposures that must process further as a single unitsingle unit

Execute calibration tasks in pipeline modeExecute calibration tasks in pipeline mode

Pass level 1b science data (pod files and uncalibrated science Pass level 1b science data (pod files and uncalibrated science datasets) and jitter files to Hubble Data Archivedatasets) and jitter files to Hubble Data Archive




4 of 24

OPUS Requirements for Thermal Vac

OPUS will develop a COS science data processing pipeline OPUS will develop a COS science data processing pipeline capable of supporting Thermal Vac testingcapable of supporting Thermal Vac testing

No database access in Thermal Vac No database access in Thermal Vac – No support schedule available in PMDB formatNo support schedule available in PMDB format– Necessary support schedule information to be read in from ASCII fileNecessary support schedule information to be read in from ASCII file

No associations No associations No calibrationNo calibration Data will be archived to HDAData will be archived to HDA Processing on Sun / Solaris UNIX platformProcessing on Sun / Solaris UNIX platform

– Need to know which version of Solaris will be used?Need to know which version of Solaris will be used?




5 of 24

OPUS Requirements for Thermal Vac (cont.)

Thermal Vac OPUS pipeline delivery scheduleThermal Vac OPUS pipeline delivery schedule– Earliest possible COS Thermal Vac currently scheduled for Earliest possible COS Thermal Vac currently scheduled for

December 10, 2001December 10, 2001– OPUS Thermal Vac pipeline due about two months prior to OPUS Thermal Vac pipeline due about two months prior to

Thermal Vac, September 24, 2000.Thermal Vac, September 24, 2000.– Beta version of OPUS pipeline due about five months prior to Beta version of OPUS pipeline due about five months prior to

Thermal Vac, June 25, 2000.Thermal Vac, June 25, 2000.




6 of 24

OPUS science data processing pipeline for COS

PM D B

PD B

KeywordD atabase

D ataP artition ing

D ataV alida tion

G enericC onversion

C alib ra tion

D ata C o llector

S upport S chedu le

W orld C oord ina teS ystem

A rch iveIn terface

KeywordExceptions

T able

KeywordR ules

pod file (leve l 1 data)

ED T dataset

ED T dataset

ED T dataset

ED T dataset

C alibrationreferencefiles and

tables

uncalibrated F IT S sc ience dataset

uncalibrated F IT S sc ience datasets

calibrated F IT S sc ience dataset




7 of 24

OPUS Processes

Data PartitioningData Partitioning– segments the telemetry stream into standard EDT datasetsegments the telemetry stream into standard EDT dataset– fill data inserted if telemetry drop-outs existfill data inserted if telemetry drop-outs exist

> event lost in Time-Tag modeevent lost in Time-Tag mode> constructs a data quality image in ACCUM mode to ensure the subsequent constructs a data quality image in ACCUM mode to ensure the subsequent

science processing does not interpret fill data as valid science datascience processing does not interpret fill data as valid science data

Support ScheduleSupport Schedule– gathers proposal information from PMDBgathers proposal information from PMDB– test proposals required for developmenttest proposals required for development

> test version of PMDB must be populated by TRANStest version of PMDB must be populated by TRANS> Thermal Vac support schedule to be input from ASCII fileThermal Vac support schedule to be input from ASCII file




8 of 24

OPUS Processes (cont.)

Data ValidationData Validation– decodes the exposure and engineering parameters in the telemetry and decodes the exposure and engineering parameters in the telemetry and

compares them to the planned valuescompares them to the planned values– internal header specification (from Ball)internal header specification (from Ball)

> PDB (EUDL.DAT, TDFD.DAT) must be fully populated and defined in PDB (EUDL.DAT, TDFD.DAT) must be fully populated and defined in DM-06 DM-06

– flags and indicators need to be determined by the Instrument Scientistsflags and indicators need to be determined by the Instrument Scientists

World Coordinate SystemWorld Coordinate System– implements a translation from telescope coordinates through the implements a translation from telescope coordinates through the

instrument light-path to an astronomically valid pointinginstrument light-path to an astronomically valid pointing– aperture positions must be definedaperture positions must be defined




9 of 24

OPUS Processes (cont.) Generic ConversionGeneric Conversion

– Generic Conversion outputs uncalibrated dataGeneric Conversion outputs uncalibrated data> rtg (Time-Tag); raw, rs1, rs2 (ACCUM); spt; trlrtg (Time-Tag); raw, rs1, rs2 (ACCUM); spt; trl

– data will be output in standard FITS format with image extensionsdata will be output in standard FITS format with image extensions– primary header will contain keywords inherited by all extensions and a primary header will contain keywords inherited by all extensions and a

null data arraynull data array– Image extensionsImage extensions

> Time-Tag mode will be FITS binary tablesTime-Tag mode will be FITS binary tables> ACCUM mode will be images grouped by imsets consisting of science ACCUM mode will be images grouped by imsets consisting of science

array and data quality arrayarray and data quality array• FUV data will contain separate imsets for each detector (a and b)FUV data will contain separate imsets for each detector (a and b)• data quality array will be null if no telemetry dropoutsdata quality array will be null if no telemetry dropouts

– calibration generates full data quality array with all other DQ flagscalibration generates full data quality array with all other DQ flags




10 of 24

OPUS Processes (cont.)

Generic Conversion (cont.)Generic Conversion (cont.)– Required for developmentRequired for development

> DM-06 to develop algorithms for data formattingDM-06 to develop algorithms for data formatting> keyword definitions (ICD-19) must be provided by the keyword definitions (ICD-19) must be provided by the

Instrument ScientistsInstrument Scientists• world coordinate definitionsworld coordinate definitions

• exposure time calculationsexposure time calculations

• calibration switches and selection criteriacalibration switches and selection criteria

• calibration file name keywordscalibration file name keywords




11 of 24

Keyword specification

– keyword namekeyword name– default valuedefault value– possible valuespossible values– unitsunits– datatypedatatype– short comment for headershort comment for header

– long descriptionlong description– header positionheader position– DADS tableDADS table– keyword sourcekeyword source

• The following information must be provided by STScI Science Instrument team for all COS specific keywords using a standard form for keyword database input.




12 of 24

Science Instrument Configuration Keywords Keyword: some possible values Keyword: some possible values

– OBSTYPE: Spectroscopy / Imaging OBSTYPE: Spectroscopy / Imaging – OBSMODE: ACCUM / TIME-TAGOBSMODE: ACCUM / TIME-TAG– EXPTYPE:EXPTYPE: ACQ,ACQ, PEAKUP/XDISP, PEAKUP/DISP, SCI,PEAKUP/XDISP, PEAKUP/DISP, SCI, WAVE,WAVE,

FLAT, DARK,FLAT, DARK, PHAPHA– DETECTOR: FUV , NUVDETECTOR: FUV , NUV– OPT_ELEM:OPT_ELEM: G130M, G160M, G185M, G225M, G285M, G140L,G130M, G160M, G185M, G225M, G285M, G140L,

G230L, MIRROR, MIRRORBG230L, MIRROR, MIRRORB– CENWAVE:CENWAVE: 1298, 1309, 1320, etc.1298, 1309, 1320, etc.– APERTURE :APERTURE : PSA, BOA, WCA, FCAPSA, BOA, WCA, FCA

(Indicates a COS unique keyword or value)(Indicates a COS unique keyword or value)




13 of 24

OPUS Processes (cont.) Data CollectorData Collector

– association table contains all information about product datasetassociation table contains all information about product dataset> dataset self-documentingdataset self-documenting

– only associations required for data processing with be constructed in the only associations required for data processing with be constructed in the OPUS pipelineOPUS pipeline

> FP splitsFP splits> Repeat obsRepeat obs

– OPUS will ensure all component exposures are present before processing OPUS will ensure all component exposures are present before processing furtherfurther

> association time-out rules need to be defined by Instrument Scientistsassociation time-out rules need to be defined by Instrument Scientists> rules for processing incomplete associations need to be defined by rules for processing incomplete associations need to be defined by

Instrument ScientistsInstrument Scientists




14 of 24

Wavecal considerations

Current plans for executing wavecalsCurrent plans for executing wavecals– Wavecal executed after each grating, central wavelength, Wavecal executed after each grating, central wavelength,

mirror, or aperture changemirror, or aperture change– Wavecal executed at least once each orbitWavecal executed at least once each orbit

Wavecal exposures will be shared between associationsWavecal exposures will be shared between associations– A single wavecal may go into multiple productsA single wavecal may go into multiple products

> For example, wavecal at start of orbit can be shared by all For example, wavecal at start of orbit can be shared by all observations in that orbit if it is applicableobservations in that orbit if it is applicable

– Adds complexity to science data processing pipelineAdds complexity to science data processing pipeline




15 of 24

Calibration

OPUS will use STSDAS calibration softwareOPUS will use STSDAS calibration software run on ALPHA / Tru64 UNIX platform in run on ALPHA / Tru64 UNIX platform in

operations operations expands size of datasetexpands size of dataset

– converts integer raw data to realconverts integer raw data to real– possible expansion of data quality array for ACCUM possible expansion of data quality array for ACCUM

modemode

Need calibration reference files for testing (at least Need calibration reference files for testing (at least dummies)dummies)




16 of 24

Other Science Data Modes

requirements for data content of each of these other requirements for data content of each of these other science data modes must be defined by Instrument science data modes must be defined by Instrument ScientistsScientists– target acquisitionstarget acquisitions– microprocessor memory dumpmicroprocessor memory dump– engineering diagnostic dataengineering diagnostic data– cumulative detector imagecumulative detector image

> MAMA cumulative image like STIS for NUVMAMA cumulative image like STIS for NUV> Possible cumulative image for FUV detectorPossible cumulative image for FUV detector




17 of 24

Engineering data processing

Receive engineering telemetry data from CCS at GSFCReceive engineering telemetry data from CCS at GSFC Process Engineering data through FGS Data PipelineProcess Engineering data through FGS Data Pipeline Generate data products to characterize jitter and Generate data products to characterize jitter and

pointing control information in support of science pointing control information in support of science observationsobservations

COS jitter file association packaging will mimic science COS jitter file association packaging will mimic science data associationsdata associations

No other COS specific requirementsNo other COS specific requirements




18 of 24

Archive Interface

OPUS will present to the archive:OPUS will present to the archive:– Original data received from PACOR (binary pod files)Original data received from PACOR (binary pod files)– Possibly pod file data packaged by observation in FITS Possibly pod file data packaged by observation in FITS

formatformat– Output of Generic Conversion (uncalibrated science Output of Generic Conversion (uncalibrated science

dataset) in FITS formatdataset) in FITS format– Jitter files in FITS format from the engineering Jitter files in FITS format from the engineering

telemetrytelemetry– Data from other science modes (target acquisition, Data from other science modes (target acquisition,

memory dump, engineering diagnostic data, cumulative memory dump, engineering diagnostic data, cumulative MAMA image) in FITS formatMAMA image) in FITS format




19 of 24

OTFR

In OTFR, data retrieved from the archive are In OTFR, data retrieved from the archive are reprocessed from the pod filereprocessed from the pod file– Provides HST data user with optimal product at time of Provides HST data user with optimal product at time of

retrievalretrieval> Calibration updates, bug fixes, and new software features and Calibration updates, bug fixes, and new software features and

algorithms available to archive usersalgorithms available to archive users

– OTFR pipeline uses the exact same code as current initial OTFR pipeline uses the exact same code as current initial science data processingscience data processing

> Reduces software development and maintenance costsReduces software development and maintenance costs> No science instrument specific code developed for OTFR beyond No science instrument specific code developed for OTFR beyond

what is necessary for initial data processingwhat is necessary for initial data processing

– Adds negligible time for retrievalsAdds negligible time for retrievals




20 of 24

OTFR

O PUS HSTScience Data Processing

P A C O R /D D F

O PU S ScienceD ata R eceip t

D A D Singest

D A D Sretrieva l

D A D Sdistribu tion

S tarV iew

A rch iveC ata log

initialprocessing OTFR

level 0 sc iencete lem etry

level 1 sc iencete lem etry (pod file )

pod file

pod file

m etadata

in itia lly ca libra teddata

pod file(s)

optim ally ca libra tedsc ience data

m etadata

optim ally ca libra tedsc ience data

data request

query

pod filefilte r

pod file

A rch iveC ata log

A rch iveS torageM edia

uncalibra teddata, pod files

HST

Archive Users




21 of 24

CODE REUSE Core OPUS system (OPUS 12.1)Core OPUS system (OPUS 12.1)

– ~236,000 lines of code~236,000 lines of code– 100% reuse100% reuse

COS specific processesCOS specific processes– Based on FUSE study (Rose et al. 1998, “OPUS: The FUSE Data Based on FUSE study (Rose et al. 1998, “OPUS: The FUSE Data

Pipeline”, Pipeline”, www.www.stscistsci..eduedu/software/OPUS/kona2.html/software/OPUS/kona2.html))> 5076 lines of code5076 lines of code> 71% reuse of existing OPUS modules71% reuse of existing OPUS modules

Expect > 99% reuse of existing data processing software for Expect > 99% reuse of existing data processing software for COS, based on lines of code.COS, based on lines of code.

– All SI complexity contained in relatively few lines of code.All SI complexity contained in relatively few lines of code. Efficient use of existing system!Efficient use of existing system!




22 of 24

Operational Considerations

No throughput issues anticipated based on COS data processing No throughput issues anticipated based on COS data processing requirements requirements

OPUS sizing based on SSR limitations in ACS eraOPUS sizing based on SSR limitations in ACS era– COS data volumes should be easily accommodated COS data volumes should be easily accommodated

Hence, Hence, OPUS pipeline throughput for COS could be handled with OPUS pipeline throughput for COS could be handled with

existing operational environmentexisting operational environment No new system requirements for running CALCOS in OPUSNo new system requirements for running CALCOS in OPUS No additional hardware specificationsNo additional hardware specifications




23 of 24

Major Science Data Processing Requirements Summary Internal header specificationInternal header specification

– DM-06DM-06 to document content and format of science internal to document content and format of science internal headerheader

– PDB (EUDL.DAT, TDFD.DAT) defined and fully populatedPDB (EUDL.DAT, TDFD.DAT) defined and fully populated

Keyword definitions Keyword definitions Flags and indicators for Data ValidationFlags and indicators for Data Validation Aperture definitionsAperture definitions




24 of 24

Test Data Requirements

Test data from detectors on optical bench expected in March Test data from detectors on optical bench expected in March 2001 and from integrated instrument in August 20012001 and from integrated instrument in August 2001

Test data to be provided by IDT/Instrument Scientists and Test data to be provided by IDT/Instrument Scientists and Engineers should include all science modesEngineers should include all science modes

Test data must includeTest data must include– PMDB population and PDB definitionPMDB population and PDB definition– spectra that enable OPUS to determine proper orientationspectra that enable OPUS to determine proper orientation– list of possible error conditions to simulate list of possible error conditions to simulate – data that simulate error conditionsdata that simulate error conditions– enough data for throughput testenough data for throughput test– engineering data to test jitter file productionengineering data to test jitter file production

COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 1 of 24 Science Data Processing

Documents