How the VO helped building the ALMA Science Archiveamiga.iaa.es/FCKeditor/UserFiles/File/VOandALMAarchive_web.pdf · Who am I Member of the AMIGA international collaboration, based

How the VO helped building the ALMA Science ArchiveJuan de Dios Santander Vela (IAA-CSIC)Felix Stoehr (ESO, ALMA Archive Scientist)

X Reunión Científica de la SEAValencia, 9 de Julio de 2012

Talk OutlineWho am I

The problem: Archiving & accessing ALMA data

The solution: ALMA Science Archive

The enabler: VO Technologies

Implementing the ASA

Conclusions & Future work

Who am IMember of the AMIGA international collaboration, based at IAA-CSIC

Ph.D. on bringing Radio Astronomical data archives and tools into the VO

Applied Scientist at ESO VLT archive, Software Engineer/Astronomy Specialist at ALMA archive (May 2009-Dec 2011)

Back to IAA-CSIC as VIA-SKA Project Manager

EXPERIENCE ON (RADIO)ASTRONOMICAL DATA,

METADATA, VIRTUAL OBSERVATORY,

ASTRONOMICAL OBSERVATIONS, ARCHIVES, AND

THE ALMA SCIENCE DATA MODEL

GROUP INTEREST IN TECH DEVELOPMENTS FOR BETTER SCIENCE

Archiving & Accessing ALMA Data

THE PROBLEM

Archiving & Accessing ALMA Data

Archiving & Accessing ALMA DataRight now (ALMA Cycle 1)

More than 32 antennas (12m) in Chajnantor

More than 9 antennas (7m) in ACA

Baselines 160m to 1km (more than 500 baselines)

Receivers for 4-bands (3,4,6,9), 8-10-12 GHz bandwidths

Archiving & Accessing ALMA DataWhen finished…

More than 50 antennas (12m) in Chajnantor

12 antennas (7m) in ACA

Around 1300 baselines, up to 18 km

Receivers for 10-bands (30-950 GHz), 8-12 GHz bandwidth, SSB/DSB

LARGE AMOUNT OF DATA AND

METADATA TO BE STORED IN ORDER

TO BE LATER PROCESSED

Archiving & Accessing ALMA DataLarge amounts of entities, data, timestamps to be stored, with high data rate

ALMA Science Data Model (Viallefond, Caillat, others) ➡ Science Data Model (ALMA, EVLA…)

XML Database implementation of CASA MS

ALMA Project Data Model

Projects, PIs, Science Goals…

calat_watererror

asdmtable asdm

almaradiometer

almar_spectralwindowid

annotation

annot_antennaid

annot_vvdvalues

annot_vvllvalue

antennaanten_offset

anten_position

beam

calampli

calam_apertureefficiency

calam_apertureefficiencyer

calam_frequencyrangecalam_polarization

types

calatmosphere

calat_forwardeffspectrum

calat_frequencyrange

calat_frequencyspectrum

calat_powerloadspectrum

calat_powerskyspectrum

calat_tatm

calat_tatmspectrum

calat_tau

calat_tauspectrum

calat_trec

calat_trecspectrum

calat_tsys

calat_tsysspectrum

calat_watercalat_alpha

spectrum

calat_forwardefficiency

calat_forwardefficiencyerr

calat_sbgain

calat_sbgainerror

calat_sbgainspectrum

calbandpass

calba_antennanames

calba_curve

calba_freqlimits

calba_polarizationtypes

calba_reducedchisquared

calba_rms

calcurve

calcu_antennanames

calcu_curve

calcu_frequencyrange

calcu_polarizationtypes

calcu_reducedchisquared

calcu_rms

caldata

calda_scanset

calda_fieldname

calda_scanintent

calda_sourcecode

calda_sourcename

caldelay

calde_delayerror

calde_delayoffset

calde_polarizationtypes

calde_reducedchisquaredcalde_ref

freq

calde_reffreqphase

calde_sidebands

caldevice

calde_caleff

calde_noisecalcalde_temperature

load

calflux

calfl_flux

calfl_fluxerror

calfl_frequencyranges

calfl_stokes

calfl_directioncalfl

_pa

calfl_paerror

calfl_sizesource

calfl_sizeerror

calfocusmodel

calfo_coefferror

calfo_coefffixed

calfo_coeffformula

calfo_coeffname

calfo_coeffvalue

calfo_focusrms

calfocus

calfo_frequencyrange

calfo_offset

calfo_offseterror

calfo_offsetwastied

calfo_pointingdirection

calfo_polarizationtypes

calfo_reducedchisquared

calfo_werefixedcalfo_focuscurve

wasfixed

calfo_focuscurvewidth

calfo_focuscurvewidtherror

calfo_offintensity

calfo_offintensityerror

calfo_peakintensity

calfo_peakintensityerror

calgain

calholography

calho_direction

calho_focusposition

calho_frequencyrange

calho_gravoptrange

calho_screwmotion

calho_screwmotionerror

calho_screwname

calho_tempoptrange

calphase

calph_ampli

calph_antennanames

calph_baselinelengths

calph_decorrelationfactor

calph_direction

calph_frequencyrange

calph_phase

calph_phaserms

calph_polarizationtypes

calph_statphaserms

calph_correctionvalidity

calpointingmodel

calpo_coefferror

calpo_coefffixedcalpo_coeff

name

calpo_coeffval

calpo_coeffformula

calpointing

calpo_collerror

calpo_colloffsetabsolute

calpo_colloffsetrelative

calpo_colloffsettied

calpo_direction

calpo_frequencyrange

calpo_polarizationtypes

calpo_reducedchisquared

calpo_beampa

calpo_beampaerror

calpo_beamwidth

calpo_beamwidtherror

calpo_beamwidthwasfixed

calpo_offintensity calpo_offinte

nsityerror

calpo_peakintensity

calpo_peakintensityerror

calposition

calpo_antennaposition

calpo_positionerr

calpo_positionoffset

calpo_refantennanames

calpo_stationposition

calprimarybeam

calpr_frequencyrange

calpr_mainbeamefficiency

calreduction

calre_appliedcalibrations

calre_paramset

calseeing

calse_baselinelengths

calse_frequencyrange

calse_phaserms

calwvr

calwv_chanfreq

calwv_chanwidth

calwv_drypath

calwv_inputantennanames

calwv_pathcoeff

calwv_polyfreqlimits calwv_r

eftemp

calwv_water

calwv_wetpath

configdescription

confi_antennaid

confi_datadescriptionidconfi_f

eedid

confi_switchcycleid

confi_assocconfigdescripti

confi_atmphasecorrection

confi_phasedarraylist

correlatormode

corre_axesorderarray

corre_basebandconfig

corre_basebandnames corre_filt

ermode

datadescription

delaymodel

delay_atmdrydelay

delay_atmwetdelay

delay_clockdelay

delay_geomdelay

delay_dispdelay

delay_groupdelay

delay_phasedelay

doppler

ephemeris

execblock

execb_antennaid

feed

feed_receiverid

feed_beamid

feed_beamoffset

feed_focusreference

feed_polresponse

feed_receptorangle

feed_illumoffset

feed_position

field

field_delaydir

field_phasedir field_refer

encedirflagcmd

focusmodel

focus_coeffformula

focus_coeffname

focus_coeffval

focus

focus_focusoffset

focus_measuredfocuspositio

freqoffset

gaintracking

gaint_cabledelay

gaint_lopropagationdelay

gaint_receiverdelay

gaint_attfreq

gaint_attspectrum

gaint_freqoffset

gaint_phaseoffset

history hologr

aphy

main

main_stateid

observation

pointingmodel

point_coeffname

point_coeffval

point_coeffformula

pointing

point_encoder

point_offset

point_pointingdirection

point_target

point_sampledtimeinterval

point_sourceoffset

polarization

polar_corrproduct

processor

receiver

recei_freqlo

recei_sidebandlo

sbsummary

sbsum_centerdirection

sbsum_observingmode

sbsum_sciencegoal

sbsum_weatherconstraint

scan

scan_caldatatype

scan_calibrationonline scan_sca

nintent

scan_calibrationset

scan_fieldname

seeing

seein_baselength

seein_phaserms

source

sourc_direction

sourc_propermotion

sourc_flux

sourc_fluxerr

sourc_frequency

sourc_frequencyinterval

sourc_position

sourc_positionangle

sourc_positionangleerr

sourc_rangevel

sourc_restfrequency

sourc_sizesource

sourc_sizeerr

sourc_stokesparameter

sourc_sysvelsourc_tr

ansition

spectralwindow

spect_assocspectralwindowi

spect_assocnature

spect_chanfreqarray

spect_chanwidtharray

spect_effectivebwarray

spect_linearray

spect_resolutionarray

squarelawdetector

state

station

stati_position

subscan

subsc_numbersubintegration

switchcycle

switc_diroffsetarray switc_freqoffset

array

switc_stepdurationarray

switc_weightarray

syscalsysca_phasediff

spectrum

sysca_tantspectrum

sysca_tanttsysspectrum

sysca_tcalspectrum

sysca_trxspectrum

sysca_tskyspectrum

sysca_tsysspectrum

totalpowertotal_

stateid

total_exposure

total_flagant

total_flagpol

total_floatdata

total_timecentroid

total_uvw

weather

wvmcal

wvmca_pathcoeff

wvmca_polyfreqlimits

wvmca_reftemp

THE ALMA SCIENCE DATA

MODEL, EXPLODED

calfl

calflsour

calfocusmodel

calfo_coefferror

calfo_coefffixed

calfo_coeffformula

calfo_coeffname

calfo_coeffvalue

calfo_focusrms

calfocus


calfo_offset

calfo_offseterror

calfo_offsetwastied





wasfixed



calfo_offintensity


calfo_peakintensity


calgain

calholography

calho_direction

calho_focusposition


calho_gravoptrange

calho_screwmotion


calho_screwname

calho_tempoptrange

calphase

calph_ampli

calph_antennanames



calph_direction


calph_phase

calph_phaserms


calph_statphaserms


calpointingmodel

calpo_coefferror


name

calpo_coeffval

calpo_coeffformula

calpo_collerror




calpo_diection

calat_watererror

calam_frequency

calatmosphere






calat_tatm

calat_tatmspectrum

calat_tau

calat_tauspectrum

calat_trec

calat_trecspectrum

calat_tsys

calat_tsysspectrum


spectrum



calat_sbgain

calat_sbgainerror


calbandpass

calba_antennanames

calba_curve

calba_freqlimits



calba_rms

calcurve

calcu_antennanames

calcu_curve




calcu_rms

caldata

calda_scanset

calda_fieldname

calda_scanintent

calda_sourcecode

calda_sourcename

caldelay

calde_delayerror

calde_delayoffset



freq

calde_reffreqphase

calde_sidebands

caldevice

calde_caleff


load

_ux

calfl_fluxerror


scan

scan_caldatatype

scan_calibrationset

scan_fieldname

seeing

seein_baselength

seein_phaserms

source

sourc_direction

sourc_propermotion

sourc_flux

sourc_fluxerr

sourc_frequency


sourc_position

sourc_positionangle


sourc_rangevel

sourc_restfrequency

sourc_sizesource

sourc_sizeerr



ansition

spectralwindow


spect_assocnature

spect_chanfreqarray



spect_linearray


squarelawdetector

state

station

stati_position

subscan


switchcycle


array


switc_weightarray

sysca_tantspectrum


feed

feed_receiverid

feed_beamid

feed_receptorangle

feed_illumoffset

feed_position

field

field_delaydir


encedirflagcmd

focusmodel

focus_coeffformula

focus_coeffname

focus_coeffval

focus

focus_focusoffset


freqoffset

gaintracking

gaint_cabledelay


gaint_receiverdelay

gaint_attfreq

gaint_attspectrum

gaint_freqoffset

gaint_phaseoffset

history hologr

aphy

main

main_stateid

observation

pointingmodel

point_coeffname

point_coeffval

point_coeffformula

pointing

point_encoder

point_offset


point_target


point_sourceoffset

polarization

polar_corrproduct

calreduction

alibrations

calre_paramset

calseeing



calse_phaserms

calwvr

calwv_chanfreq

calwv_chanwidth

calwv_drypath


calwv_pathcoeff


eftemp

calwv_water

calwv_wetpath

configdescription

confi_antennaid


eedid

confi_switchcycleid




correlatormode




ermode

calfl

calflsour

calfocusmodel

calfo_coefferror

calfo_coefffixed

calfo_coeffformula

calfo_coeffname

calfo_coeffvalue

calfo_focusrms

calfocus


calfo_offset

calfo_offseterror

calfo_offsetwastied





wasfixed



calfo_offintensity


calfo_peakintensity


calgain

calholography

calho_direction

calho_focusposition


calho_gravoptrange

calho_screwmotion


calho_screwname

calho_tempoptrange

calphase

calph_ampli

calph_antennanames



calph_direction


calph_phase

calph_phaserms


calph_statphaserms


calpointingmodel

calpo_coefferror


name

calpo_coeffval

calpo_coeffformula

calpo_collerror




calpo_diection

calat_watererror

asdmtable asdm

almaradiometer

almar_spectralwindowid

annotation

annot_antennaid

annot_vvdvalues

annot_vvllvalue

antennaanten_offset

anten_position

beam

calampli

calam_apertureefficiency

calam_apertureefficiencyer

calam_frequencyrangecalam_polarization

types

calatmosphere






calat_tatm

calat_tatmspectrum

calat_tau

calat_tauspectrum

calat_trec

calat_trecspectrum

calat_tsys

calat_tsysspectrum


spectrum



calat_sbgain

calat_sbgainerror


calbandpass

calba_antennanames

calba_curve

calba_freqlimits



calba_rms

calcurve

calcu_antennanames

calcu_curve




calcu_rms

caldata

calda_scanset

calda_fieldname

calda_scanintent

calda_sourcecode

calda_sourcename

caldelay

calde_delayerror

calde_delayoffset



freq

calde_reffreqphase

calde_sidebands

caldevice

calde_caleff


load

calflux

calfl_flux

calfl_fluxerror


calfl_stokes

calfl_directioncalfl

_pa

calfl_paerror

calfl_sizesource

calfl_sizeerror

calfocusmodel

calfo_coefferror

calfo_coefffixed

calfo_coeffformula

calfo_coeffname

calfo_coeffvalue

calfo_focusrms

calfocus


calfo_offset

calfo_offseterror

calfo_offsetwastied





wasfixed



calfo_offintensity


calfo_peakintensity


calgain

calholography

calho_direction

calho_focusposition


calho_gravoptrange

calho_screwmotion


calho_screwname

calho_tempoptrange

calphase

calph_ampli

calph_antennanames



calph_direction


calph_phase

calph_phaserms


calph_statphaserms


calpointingmodel

calpo_coefferror


name

calpo_coeffval

calpo_coeffformula

calpointing

calpo_collerror




calpo_direction

calpo_frequencyrange

calpo_polarizationtypes

calpo_reducedchisquared

calpo_beampa

calpo_beampaerror

calpo_beamwidth

calpo_beamwidtherror

calpo_beamwidthwasfixed

calpo_offintensity calpo_offinte

nsityerror

calpo_peakintensity

calpo_peakintensityerror

calposition

calpo_antennaposition

calpo_positionerr

calpo_positionoffset

calpo_refantennanames

calpo_stationposition

calprimarybeam

calpr_frequencyrange

calpr_mainbeamefficiency

calreduction

calre_appliedcalibrations

calre_paramset

calseeing



calse_phaserms

calwvr

calwv_chanfreq

calwv_chanwidth

calwv_drypath


calwv_pathcoeff


eftemp

calwv_water

calwv_wetpath

configdescription

confi_antennaid


eedid

confi_switchcycleid




correlatormode




ermode

datadescription

delaymodel

delay_atmdrydelay

delay_atmwetdelay

delay_clockdelay

delay_geomdelay

delay_dispdelay

delay_groupdelay

delay_phasedelay

doppler

ephemeris

execblock

execb_antennaid

feed

feed_receiverid

feed_beamid

feed_beamoffset

feed_focusreference

feed_polresponse

feed_receptorangle

feed_illumoffset

feed_position

field

field_delaydir


encedirflagcmd

focusmodel

focus_coeffformula

focus_coeffname

focus_coeffval

focus

focus_focusoffset


freqoffset

gaintracking

gaint_cabledelay


gaint_receiverdelay

gaint_attfreq

gaint_attspectrum

gaint_freqoffset

gaint_phaseoffset

history hologr

aphy

main

main_stateid

observation

pointingmodel

point_coeffname

point_coeffval

point_coeffformula

pointing

point_encoder

point_offset


point_target


point_sourceoffset

polarization

polar_corrproduct

processor

receiver

recei_freqlo

recei_sidebandlo

sbsummary

sbsum_centerdirection

sbsum_observingmode

sbsum_sciencegoal

sbsum_weatherconstraint

scan

scan_caldatatype

scan_calibrationonline scan_sca

nintent

scan_calibrationset

scan_fieldname

seeing

seein_baselength

seein_phaserms

source

sourc_direction

sourc_propermotion

sourc_flux

sourc_fluxerr

sourc_frequency


sourc_position

sourc_positionangle


sourc_rangevel

sourc_restfrequency

sourc_sizesource

sourc_sizeerr



ansition

spectralwindow


spect_assocnature

spect_chanfreqarray



spect_linearray


squarelawdetector

state

station

stati_position

subscan


switchcycle


array


switc_weightarray

syscalsysca_phasediff

spectrum

sysca_tantspectrum


sysca_tcalspectrum

sysca_trxspectrum

sysca_tskyspectrum

sysca_tsysspectrum

totalpowertotal_

stateid

total_exposure

total_flagant

total_flagpol

total_floatdata

total_timecentroid

total_uvw

weather

wvmcal

wvmca_pathcoeff

wvmca_polyfreqlimits

wvmca_reftemp

MORE THAN 360 ASDM

TABLES & SUBTABLES!

Archiving & Accessing ALMA DataASDM, APDM are CASA-friendly, not astronomer/science friendly

ALMA Frontend Archive, optimized for storage & preservation, not for data query/retrieval

We need an ALMA Science Archive, at least for when there are science data to be released

We cannot build it on top of the ASDM

directly

ALMA Science ArchiveTHE SOLUTION

ALMA Science Archive


AFADB

ASADB harvesting

ASA IF

queries


AFADB

ASADB harvesting

ASA IF

queries

No changes to AFA DB

Harvesting takes selected datasets from AFA DB into ASA DB

ASA DB consists of scientific relevant parameters

ASA IF supports the different clients: engineers, scientists, and the VO

ASA Elements: HarvesterLightweight process. Current ALMA holdings can be reprocessed in less than 4 hours

Ensures only correct data enters the ASA

Performs tasks such as:ASDM parsing & consolidationASDM consistency checkscoordinate conversionsolar system body identificationbaseline reprojection

harvesting

ASA Elements: ASA DBPlain relational SQL DB

Based on Hubble Legacy Archive experience

Denormalized StructureScienceProjectSpectralWindowsProvenance

Supports pipeline reduced and externally reduced visibilities & data products

Allows joint querying of all product types

ASADB

INSPIRED BY IVOA

OBSCORE DATA MODEL,

ASDM

ASA Elements: InterfacesASA Interfaces built as web application

VO metadata-driven

Clients supported:QA/CSV: technical parameters (scheduling blocks, scan intents)Scientists: science parameters

Resolution (spatial, frequency & velocity)Positional and target searches

Programatic/VO interface

ASA IF

VO Technologies for the ALMA Archive

THE ENABLER

VO Technologies for the ALMA ArchiveVirtual Observatory: federation of archives sharing a set of common practices and data models that allow for easy discoverability of interoperable data-sets, with unified description by means of a common data model, within the realm of astronomy.

Data model: Description of the set of entities needed for information storage in a particular field, specifying both the data being stored, and the relationships between them.

NEED OF SOFTWARE TO

IMPLEMENT IT

VO Technologies for the ALMA ArchiveVO Data Models

ASA DB structure inspired from ObsCore, RADAMS, Hubble Legacy Archive

VO Software

openCADC (DB access, VO access protocols, metadata management)

VOView (VOTable rendering web component)

MAXIMISATION OF SOFTWARE REUSE,MAXIMISATION OF OUTPUT PER DEVELOPER

⤴

ASA ImplementationTHE RESULT

ASDM Harvester

RDBMS

ASA Tables

openCADC

Tomcat Web App VO API

VOView

ASA ImplementationVO ENABLED!

ASDM Harvester

RDBMS

ASA Tables

openCADC

Tomcat Web App VO API

VOView

ASA Implementation

JUST NEED TO WRITE THESE

AND CONNECTORS!

Demo

Conclusions & Future Work

Conclusions & Future WorkThe separation of the ASA & AFA allows for different parameter spaces to be queried (instrumental vs. scientific) on the ALMA archive

Using IVOA standards for metadata allows us to use VO solutions to implement a science archive

And reduce the manpower to implement it!

VO software is mature enough to implement a AND YOU GET VO COMPATIBILITY FOR FREE!


Interface being simplified, even more VO metadata driven

Expected to go public by the end of 2012, when the first reduced project data end their proprietary periods


Home -> Archive query -> Science query

Science Query

Search Reset

Position

Source name (Sesame)Source name (ALMA)RA DecSearch radius

00:10:00

Energy

BandFrequencyBandwidthSpectral resolutionChannels

Time

Observation dateIntegration time

Polarization

Polarization type

Observation

Project codeWater vaporScan intentScheduling Block namePad/Antenna name

Query Form Result Table Get Data

FUTURE INTERFACE

MOCKUP

Gràcies!

¡Gracias!

Thanks!

Questions?

How the VO helped building the ALMA Science Archiveamiga.iaa.es/FCKeditor/UserFiles/File/VOandALMAarchive_web.pdf · Who am I Member of the AMIGA international collaboration, based

Documents