HMI-AIA JSOC Science Data Processing (SDP) Readiness Art Amezcua, JSOC Software Lead. AGENDA JSOC-SDP Overview JSOC-SDP Status (H/W, S/W) Pipeline processing Database Level-0, Level-1 and higher levels AIA Visualization Center JSOC-SDP Maintenance and CM Documentation Staffing - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
• JSOC-SDP supports both HMI and AIA through Level-1 and HMI through Level-2 science data products
• JSO-SDP Infrastructure is complete– JSOC-SDP Hardware is complete, upgrades in process
• Database systems – Warm standby system online in August 2009
• Web Server – Upgrade online in August 2009
– JSOC-SDP Software• Data Record Management System and Storage Unit Management System (DRMS/
SUMS) complete as of March 2009
– JSOC-SDP Archive System is fully operational
• Software Components needed to support commissioning– Level-0 Image processing for both AIA and HMI is ready and was used to
support Observatory I&T– Level-0 HK, FDS and other metadata merge – complete as of May 2009– Level-1 (science observables) – will be completed during commissioning
• Software components needed to support science mission: – Production Pipeline Manager – In development, expected during commissioning– HMI Level-2 (Version 1 of science data products)
• Local Helioseismology – Work in parallel on “rings”, “time-distance”, and “holography” proceeding with basic capability, expected to be ready during commissioning
• Global Helioseismology – Ready for testing during commissioning
• Magnetic Field standard products – Ready for testing during commissioning
• Vector Field disambiguation – 80% complete with preliminary product ready by end of commissioning (requires real data to proceed)
– Export and Catalog Browse Tools • Functional but needs work (http://jsoc.stanford.edu/ajax/lookdata.html)
• Refinements will continue
– All science products need flight data during commissioning to complete development
• AIA Visualization Center (AVC) at Lockheed Martin– Higher-level AIA processing and science product generation– Heliophysics Event Knowledgebase (HEK)
• Stanford Summary: On schedule for L – 4 and Phase E – 6 months
• All machines fully tested for integrity with simulated data at realistic data rate and volumes
• All machines/components under service warranties with vendors
• Database machines have been online for four years (during DRMS development)
• Data-processing cluster, file- and tape-server machine, T950 tape robot, and tape systems went online in July 2008
• Upgrades (new machines onsite)– MOC web-access machine– solarport (gateway to SDP)– ftp server– web server– two database machines– in service in October 2009
• Higher-Level Science Products– Internal Rotation Ω(r,Θ) [estimated complete as of August 2009]– Internal Sound Speed cs(r,Θ) [estimated complete as of August 2009]– Full-Disk Velocity v(r,Θ,Φ) [estimated complete as of December 2009]– Sound Speed cs(r,Θ,Φ) [estimated complete as of December 2009]– Carrington Synoptic Velocity Maps [estimated complete as of December 2009]– Carrington Synoptic Speed Maps [estimated complete as of December 2009]– High-Resolution Velocity Maps [estimated complete as of December 2009]– High-Resolution Speed Maps [estimated complete as of December 2009]– Deep Focus Maps [estimated complete as of July 2010]– Far-Side Activity Maps [estimated complete as of December 2009]– Line-of-Sight Magnetic Field Maps [COMPLETE as of July 2009]– Vector field inversion and direction disambiguation [estimated complete as of
March 2010]– Vector Magnetic Field Maps [estimated complete as of April 2010]– Coronal Magnetic Field Extrapolations [COMPLETE as of July 2009]– Coronal and Solar Wind Models [estimated complete as of April 2010]– Brightness Images [estimated complete as of August 2009]
• Scope– AIA – Level-0 and Level-1 data– HMI – Level-0 data through Level-2 data
• Web Export– http://jsoc.stanford.edu/ajax/lookdata.html– Query for desired data, then download via web– Supports several data formats (internal files, FITS files, tar files, compressed files)– Provides support for special processing (such as extracting regions)– Other developers can expand on this export method by writing javascript that is allowed
to access our web cgi programs– Functional now; enhancements estimated complete as of August 2009
• NetDRMS– Network of DRMS sites– Can share DRMS data (not just data files) among sites using RemoteDRMS and
RemoteSUMS– Scientists can request the same data from one of many sites– Functional now; enhancements estimated complete as of August 2009
• Virtual Solar Observatory (VSO) Integration– Provides UI that allows uniform search of disparate types of data– Obtains metadata and data files from NetDRMS sites experiencing the least congestion– Estimated complete as of December 2009
• Software Version Control System with CVS– All software controlled in CVS; development occurs in “sandboxes”
• Daily Build– All binaries are compiled daily from CVS repository
• JSOC Software Release– Every month stable binaries are compiled in /home/jsoc; production processes
must use these binaries
• Change Control Board (CCB)– Data Capture July 2009– DRMS/SUMS September 2009– Pipeline Dataflow Management System December 2009– Pipeline Control Tables December 2009– Level-0 Processing July 2009– Level-1 Processing September 2009– Science Analysis Pipeline Modules no CCB
• Data Capture– OC3-line failure – there are two lines; use the second one for both AIA and HMI
– DCS-machine failure – use spare in Data Center (assumes failed DCS machine’s IP address); spare has database backup
– To mitigate media failure, there are two tape copies of tlm data, one at Stanford and one at LMSAL, and Level-0 data are archived to tape; thus there are three copies of the images.
– Catastrophic Data Capture failure exceeding 20 days – use spare DCS at MOC
– Software logging reports anomalies
• Level-0 and Level-1– If a dedicated cluster node fails, dedicate another
– Software logging, email notifications, and monitoring web pages report anomalies
• Wiki– http://jsoc.stanford.edu/jsocwiki– DRMS/SUMS overview and description– DRMS and SUMS users’ guide and developers’ guide– Release notes
• Doxygen– Manual describing DRMS API functions and modules– Provides synopsis and describes input parameters, output, return values– To date, ~ 1/2 of functions/modules have documentation
• Flow Diagrams– Tree diagrams illustrating connections between various programs, data series, data tables, etc. – Diagrammatic view of pipeline processes links to documentation– Note stages of development (A – E) and estimated completion date
• The JSOC-SDP can support:– Archive and distribution functions now– Analysis for instrument commissioning now– Initial science data processing by launch
So lets get SDO in the sky so we can have real data!
• Stage A – Code specification exists, but working code does not exist• Stage B – Prototype code exists, but not necessarily on HMI data and not
necessarily in the correct language• Stage C – Working code exists but cannot run inside JSOC pipeline• Stage D – Working code capable of running in JSOC pipeline, but undergoing
final testing and not released for general use• Stage E – Working code complete and integrated into JSOC pipeline
Following dataflow charts show status as of FORR with estimated months to complete to stage E shown