SPACEINN -‐-‐-‐-‐-‐-‐
SDO/HMI Data Base
Laurent Gizon, Ray Burston 21 October 2013, Paris
SPACEINN(helioseismology) at MPS
• Kaori Nagashima (SPACEINN MPS coordinator) >Experience with SDO and Hinode data
• Raymond Burston (MPS data base) >Experience with netDRMS, SDO formats
• Science support: Aaron Birch (ex-‐NASA SDO Science Center), Jesper Schou (HMI Project ScienUst), Hannah Schunker (ex-‐HELAS)
SDO/HMI data
• Global and local helioseismology applicaUons • All SDO data = 2 TB/day • SDO/HMI Lev.1 data = ca. 50 TB/year • Calibrated data produced by JSOC • SDO/HMI data automaUcally flowing into German Data Center (GDC)
• Access to SDO/HMI for SPACEINN users: Through SPACEINN local helioseismology website or via netDRMS
Access #1 to SDO/HMI data SPACEINN local helioseismology website
www.mps.mpg.de/projects/seismo/SpaceInn/ • Data download: sample data sets
> science ready tracked data cubes > more on request
• InformaUon on data reducUon etc. • Links: SDO-‐HMI (hmi.stanford.edu/) • Links: JSOC (jsoc.stanford.edu/) > data export • Links: Lockheed Heliophysics Knowledgebase (HEK) • Need to link this work to other SPACEINN sites
Access #2: netDRMS (JSOC Data Record Management System)
• Ray Burston will explain… • netDRMS must be installed locally • Data requests are forwarded to relevant data centers (GDC is Europe’s site for HMI)
• Upon request (must be jusUfied), GDC can push some data series automaUcally to remote sites (e.g. KIS)
SDO/HMI Data in DRMS
Doppler Velocity
Heliographic Doppler velocity
maps
Tracked Tiles Of Dopplergrams
Stokes I,V
Filtergrams
ConUnuum Brightness
Tracked full-‐disk 1-‐hour averaged ConUnuum maps
Brightness feature maps
Solar limb parameters
Stokes I,Q,U,V
Full-‐disk 10-‐min Averaged maps
Tracked Tiles
Line-‐of-‐sight Magnetograms
Vector Magnetograms Fast algorithm
Vector Magnetograms Inversion algorithm
Egression and Ingression maps
Time-‐distance Cross-‐covariance
funcUon
Ring diagrams
Wave phase shig maps
Wave travel Umes
Local wave frequency shigs
Spherical Harmonic Time series To l=1000
Mode frequencies And splihng
Brightness Images
Line-‐of-‐Sight MagneUc Field Maps
Coronal magneUc Field ExtrapolaUons
Coronal and Solar wind models
Far-‐side acUvity index
Deep-‐focus v and cs maps (0-‐200Mm)
High-‐resoluUon v and cs maps (0-‐30Mm)
Carrington synopUc v and cs maps (0-‐30Mm)
Full-‐disk velocity, v(r,Θ,Φ), And sound speed, cs(r,Θ,Φ),
Maps (0-‐30Mm)
Internal sound speed, cs(r,Θ) (0<r<R)
Internal rotaUon Ω(r,Θ) (0<r<R)
Vector MagneUc Field Maps
HMI Data Data Product Processing
Level-0
Level-1
Something to consider • netDRMS can potenUally be used as common SPACEINN
database sogware used for sharing all helio-‐ and asteroseismology data sets?
• Advantage: already in place at MPS, KIS, IAS • netDRMS excellent to share data among sites (soluUon
chosen for Plato data center) • However, ingesUng new data series = work
Next: Ray Burston
German Science Center for SDO – Dedicated data and scienUfic analysis center for SDO/HMI data – Consists of two components
GDC-‐SDO: German Data Center for SDO – Funded by the German Aerospace Agency (DLR), 2007 – 2016 – Collects SDO/HMI data directly from JSOC (USA) and archives locally at MPS – GDC-‐SDO is a master European data distribuUon center for HMI data – Storage capacity: 192 TB online + 180 TB in near-‐line tape libraries (224 slots) – To be expanded to > 500 TB online – Main data products include all HMI Dopplergrams, l.o.s. magnetograms and
conUnuum intensity images = 125 TBs
MPS compu<ng resources • Local access to SDO/HMI data via GDC-‐SDO (InfiniBand connecUvity) • ERC SISI Compute cluster: 256 cores, 2 TB RAM • To be extended to > 1000 cores • Pegasus Workflow Managment System
NetDRMS
R Bogart (2007) Astron. Nachr.328, No 3/4, 352-‐355
NetDRMS Site 3
• The Data Record Management System (NetDRMS) is developed by Stanford University with the aim of sharing SDO data worldwide
• A PostgreSQL database stores meta-‐data, e.g. keyword name/value pairs in FITS headers
• Storage Unit Management System stores/retrieves physical data (e.g. data arrays in FITS files)
• Java Mirroring Daemon transfers the HMI images using high-‐performance scp
• NetDRMS provides uUliUes for subscribing to a data series
• SLONY-‐I is required publishing data series
NetDRMS
§ NetDRMS API consists of 250 funcUons to communicate with NetDRMS from C
§ Command line uUliUes are useful for searching your NetDRMS database
§ UUliUes are also useful to obtain info in languages other than C; IDL example below
NetDRMS
GDC-‐SDO Help Desk for SPACEINN
• Located hxp://www.mps.mpg.de/projects/seismo/GDC-‐SDO/spaceinn_access.htm
• Contains informaUon for sehng up NetDRMS and receiving relevant data
• For user support, contact Ray Burston at [email protected]
Three steps for implemen<ng NetDRMS at your site
• InstrucUons for installing NetDRMS are located jsoc.stanford.edu/netdrms, and includes details on sehng up PostgreSQL DB and subscribing to published data series
• InstrucUons for installing the Java Mirroring Daemon (to transfer images) are located hxp://vso.tuc.noao.edu/VSO/w/index.php/Main_Page
• InstrucUons for publishing data series are located jsoc.stanford.edu/jsocwiki/PublishSeries
NetDRMS
Objec<ves
• Serve the community with HMI observaUons through the German Science Center for the Solar Dynamics Observatory (GSC-‐SDO)
• Make relevant HMI data derivaUves arising from scienUfic working groups, as well as complementary data (e.g. GONG, MDI, JSOC pipeline science products, GSC-‐SDO pipelines, etc.), available via the GSC-‐SDO
• Facilitate effortless and transparent data handling, thereby increasing the Ume available for innovaUve scienUfic data analysis.
WP 3.5 HMI-‐SDO Data Handling and DistribuUon
Tasks
• IdenUfy community requirements
• Maintain the Data Record Management System (NetDRMS) at GSC-‐SDO
• Provide technical support for NetDRMS installaUon/maintenance at other sites
• Monitor HMI data flow; check consistency with JSOC; ensure data integrity
• Backup and secure HMI data, science derivaUves, and complimentary data
• Create website for the disseminaUon of informaUon relevant to this project e.g. data access instrucUons
• Gather relevant informaUon from the scienUfic community and adverUse to a much wider audience through the website
• Establish help desk and user support infrastructure
Status
• All tasks have commenced and they will require conUnuous axenUon or updates
• Provided support to KIS for installing NetDRMS: interfaces are operaUonal
WP 3.5 HMI-‐SDO Data Handling and DistribuUon
Planning, Needs and Ac<on Items
• GDC-‐SDO will publish addiUonal relevant data series if required by the community
• Calculate staUsUcs on data usage and data flow precisely and automaUcally
Relevant Deliverables
• D3.16 1st report on help-‐desk and user-‐support operaUons (SDO): 12 months
• D3.19 1st report on HMI data products and usage staUsUcs: 12 months
WP 3.5 HMI-‐SDO Data Handling and DistribuUon
HMI/AIA Data Volumes