APOGEE-2 Data Infrastructure Jon Holtzman (NMSU) APOGEE team.

APOGEE-2 Data Infrastructure

Jon Holtzman (NMSU)APOGEE team

– Data infrastructure for APOGEE-2 will be similar to that of APOGEE-1, generalized to multiple observatories, and with improved tracking of processing

– APOGEE raw data and data products are stored on the Science Archive Server (SAS)

– Reduction and analysis software is (mostly) managed through the SDSS SVN repository

– Raw and reduced data described (mostly) through SDSS datamodel

– Data and processing documented via SDSS web pages and technical papers

Data infrastructure

– APOGEE instrument reads continuously (every ~10s) as data are accumulating, 3 chips at 2048x2048 each • Raw data are stored on instrument control computer (current

capacity is several weeks of data)• Individual readouts are “annotated” with information from

telescope and stored on “analysis” computer (current capacity is several months). These frames are archived to local disks that are “shelved” at APO (currently 20 x 3TB disks)

– “quick reduction” software at observatory assembles data into data cubes and compresses (lossless) for archiving on SAS• Maximum daily compressed data volume ~ 60 Gb

Raw data

Raw data

Does not include NMSU 1m + APOGEE dataLCO data will be concurrentTotal 2.5m raw data to date: ~11 TB

• “quick reduction” software estimates S/N (at H=12.2) which is inserted into plate database for use with autoscheduling decisions

• APOGEE-1– Data transferred to SAS next day, transferred to NMSU

later that day, processed with full pipeline following day, updated S/N loaded into platedb, initial QA inspection

• APOGEE-2 proposal:– Process data at observatory with full pipeline next

day, or at SAS location (Utah) and/or– Improve “quick reduction” S/N

Initial processing

• Three main stages (+1 post-processing)– APRED : processing of individual visits (multiple exposures at

different detector spectral dither positions) into visit-combined spectra, with initial RV estimates. Can be done daily

– APSTAR: combine multiple visits into combined spectra, with final RV determination. • For APOGEE-1, has been run annually (DR10: year 1, DR11: year

1+year2)

– ASPCAP: process combined (or resampled visit) spectra through stellar parameters and chemical abundances pipeline• For APOGEE-1, has been run 3 times

– ASPCAP/RESULTS: apply calibration relations to derived parameters, set flag values for these

Pipeline processing

• Raw data: data cubes (apR)• Processed exposures (maybe not of general interest?)

– 2D images (ap2D)– Extracted spectra (ap1D)– Sky subtracted and telluric corrected (apCframe)

• Visit spectra– Combine multiple exposures at different dither positions– apVisit files: native wavelength scale, but with wavelength array

• Combined spectra– Combine multiple visits, requires relative RVs– apStar files: resampled spectra to log(lambda) scale

• Derived products from spectra– Radial velocities and scatter from multiple measurements (done during combination)– Stellar parameters/chemical abundances from best-fitting template

• Parameters: Teff, log g, microturbulence (fixed), [M/H], [alpha/M], [C/M], [N/M]• Abundances for 15 individual elements

– aspcapStar and aspcapField files: stellar parameters of best-fit, pseudo-continuum normalized spectra and best fiitting templates

• Wrap-up catalog files (allStar, allVisit)

APOGEE data products

APOGEE data volumeRaw data: • 2.5m+APOGEE: ~4 TB/year APOGEE-1 ~6 TB/year with

MaNGA co-observing• 1m+APOGEE: ~2 TB/year• LCO+APOGEE: ~3 TB / yearTOTAL APOGEE-1 + APOGEE-2 : ~75 TB

Processed visit files: ~ 3 TB/year (80% individual exposure reductions)Processed combined star files: ~500 GB/100,000 starsProcessed ASPCAP files: raw FERRE files ~500 GB/100,000 starsBundled output: ~100 GB / 100,000 starsTOTAL APOGEE-1 + APOGEE-2 (one reduction!): ~ 40 TB

APOGEE data access“Flat files” available via SDSS SAS:

all intermediate and final data product filessummary ``wrap-up” files (catalog)

“Catalog files” available via SDSS CAS:apogeeVisit, apogeeStar, aspcapStar

Spectrum files available via SDSS API and web interface

Planning 4 data releases in SDSS-IV:DR14: July 2017 (data through July 2016)DR15: July 2018 (data through July 2017 – first APOGEE-S)DR16: July 2019 (data through July 2018)DR17: Dec 2020 (all data)

APOGEE software products

• apogeereduce: IDL reduction routines (apred and apstar)

• aspcap• speclib: management of spectral libraries, but not all

input software (no stellar atmospheres code, limited spectral synthesis code)

• ferre: F95 code to interpolate in libraries, find best fit• idlwrap: IDL code to manage ASPCAP processing

• apogeetarget: IDL code for targetting

APOGEE pipeline processing

• Software all installed and running on Utah servers• Software already in pipeline form (few lines per full

reduction step to distribute and complete among multiple machines/processors)• Some need to improve distribution of knowledge

and operation among team• Some external data/software required for ASPCAP

operation• Generation of stellar atmospheres (Kurucz and/or

MARCS)• Generation of synthetic spectra (ASSET, but

considering MOOG and TURBOSPECTRUM)

APOGEE software/personnel

• apogeereduce• developer: Nidever, Holtzman, (Nguyen)• operation: Holtzman, (Hayden, Nidever, Nguyen)

• ASPCAP• grids:

• ASSET: Allende Prieto / Koesterke• Turbospec: Zamora, Garcia-Hernandez, Sobeck, Garcia-

Perez, Holtzman• MOOG: Shetrone, Holtzman (pipeline), others

• speclib• postprocessing: Allende-Prieto, Holtzman

• ferre: Allende Prieto• idlwrap: Holtzman, Garcia-Perez (Shane)• Operation: Holtzman (Shane, Shetrone)

END

Abundances of cooler stars

Second instrument or first instrument relocation

Surface gravity issues: red clump vs red giant

Abundance analysis of faint bulge stars: RR Lyr and RC stars

Achieving distance distribution