WSDC Operations - California Institute of Technologywise2.ipac.caltech.edu/staff/roc/reviews/orr/30-WSDC... · 2009. 11. 16. · WSDC Operations R. Cutri, R. Beck, T. Conrow - IPAC
Post on 28-Jan-2021
0 Views
Preview:
Transcript
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
1
WSDC Operations
R. Cutri, R. Beck, T. Conrow - IPAC
October 7-8, 2009
Operational Readiness Review – October 7-8, 2009
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
2
WISE Science Data Processing Flow
Operational Readiness Review – October 7-8, 2009
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
3
Operations Procedures
• Documented in WSDC D-C001 through C005 • All in mature draft stage • Final pre-launch versions in the next month • Update procedures as necessary post-launch in response to
on-orbit data needs
Operational Readiness Review – October 7-8, 2009
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
4 Operational Readiness Review – October 7-8, 2009
• raw_copy script on Ingest computer (WSDCIN) checks filenames, sizes and creates a manifest
• get_wsdcin script on the WSDS checks for new manifests and copies over new data.
• write multiple archives to tape of MOS/HRP data.
• ingestpipe for MOS/HRP copies sequence products, CK to archive location, loads HK. HRP ingest decommutates science telemetry, creates L0, QL and scans dirs and starts QL.
Data Transfer/Ingest Procedure
MOS data
HRP data
dmz
WSDS MOS
WSDS HRP
Ingest
L0 data
Quicklook
Ingest
dmz
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
5 Operational Readiness Review – October 7-8, 2009
Scan Frame pipeline
L1 images
Multi Frame pipeline
L3 source lists L3 images
L1 source lists
L0 data
• run_scans script starts scan frame pipelines, monitors progress and creates summary when complete. • showme_summary script shows same as operator console.
• wmcpipe creates wmfpipe commands to create coadds for completed scan frame areas. • monitor_coadds script does just that.
Scan Frame/Multi Frame Processing Procedures
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
6 Operational Readiness Review – October 7-8, 2009
IMGPrep IMGPrep IMGPrep IMGPrep
IRSA L1 metadata
IRSA L3 metadata
L1 images L3 images
IMGPrep IMGPrep IMGPrep DBPrep
IRSA L1 lists
L3 source lists
L1 source lists
IRSA L3 lists
• dbprep script takes L1 or L3 source lists and creates pipe-delimited database load files
• imgprep script takes L1 or L3 images and creates pipe-delimited database load files.
DB Load Preparation Procedure
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
7 Operational Readiness Review – October 7-8, 2009
NODES Maintenance Procedures
Monitors cluster performance and disk space usage and overall performance of the cluster.
- nodes script - allows single command to be executed on all nodes - good for checking space, number of processes running, starting background processes
- node_clean script - ensures nodes do not run out of work space on their local disks - creates logs per node for what gets deleted
- condor commands - starts, stops, delete running or queued processes - monitor running processes, queued processes, owner of processes
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
8
Daily Processing Operations
• Scheduled time summary
– Minimal required ops support Daily run: 4 hr In/QL (in parallel) + 4 hr SF + 4 hr backup Elapsed: 8 hrs
– Normal ops Daily run: 4 hr In/QL (||) + 4 hr SF + 2 hr MS + 2 hr Coadd + 4 hr backup Elapsed: 12 hours
– Catch up or rerun Daily run: 4 hr In/QL (||) + 4 hr SF + 2 hr MS + 2 hr coadd + 4 hr backup Catch up: 4 hr SF + 4 hr MS (or other rerun combinations) Elapsed: 20 hours
Operational Readiness Review – October 7-8, 2009
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
9
Daily Processing Operations
Operational Readiness Review – October 7-8, 2009
PST Activity
09
10
11
12
13
14
15
16
17
18
19
20 21
22
23
00
01
0 2
03 04
05
06
07
08
HRP Xfers (auto)
T1
T2
T3
T4
Ingest/QL (auto)
InQL 3
InQL 4
InQL 1
InQL 2
Scan/frame (T+0)
S/F 1,2
S/F 3,4
Backups Backups
Multi-scan (T+1 days)
MS 1-4
Coadds (T+2 days)
Coad 1-4
Reruns, Make-up
Idle Idle
Min
. Ops
Sup
port
N
orm
al O
ps
Staffed Unstaffed
Daily Ops Processing
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
10
Daily Processing Operations Personnel Deployment
• 2 FTE net Ops Engineers; 1 FTE + 0.5 FTE + 0.5 FTE; occasional back-up from 2 other WSDC engineers
• 6 day coverage (Mon.-Sat.), staggered 4-8 hr work shifts • Daily duties and time allocations (M-F, reduced list on Sat.)
– Monitor data arrival and automated transfer – 1 hour – Confirm normal automated Ingest/Quicklook runs – 1 hour – Start Scan/frame runs for recent data – 1 hour – Start Multi-scan, coadds, WMOPS for completed scans – 2 hours
• Including scheduling overnight runs – Telem. and Ops DB backups – 2 hours – Disk, run time monitoring and clean-up – 1 hour – Handle special-purpose deliveries, reruns – 2 hours – Archive loading – 1 hour – Misc. reports – 1 hour – Total: 12 hours
Operational Readiness Review – October 7-8, 2009
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
11
QA Ops Staff Deployment IOC
Operational Readiness Review – October 7-8, 2009
• Assumptions − Net 5 FTEs available − 7 day coverage − Staggered 9h shifts
• During critical events (boresight alignment, scan rate calib) stretch daily coverage as necessary
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
12
QA Ops Staff Deployment Survey Ops
Operational Readiness Review – October 7-8, 2009
• Assumptions − Net 5 FTEs available − 5day coverage − 9h workdays (8h + 1h
meal)
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
13
Visiting Team Member Support
• Approximately 10 project team members at IPAC during IOC and 1-2 months of survey operations
• Facilities – Project “Situation Room” with work stations, dedicated meeting facility (MR214;
reconfiguration underway ) – Temporary visitor offices for additional guests
• Visitors will supply own computers – IPAC ISG will provide user support – Visitors will scp processed data to their computers for analysis
• Connectivity – Hardline network connection in Situation Room to WSDC computer to download
raw or processed data – Wireless network for general Internet access – Small volume data transfers via Ground Test Data Archive sftp site for SDL
Operational Readiness Review – October 7-8, 2009
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
14 Operational Readiness Review – October 7-8, 2009
Key Remaining Tasks Before Launch
• Complete script to archive raw HRP and MOS TLM to tape − 3 copies generated: 1 off-site in deep archive
• Finalize Multi-Scan pipeline procedure
• Finalize Pipeline Operations interactions and reporting formats with MOS (following ingest) and internal WSDC subsystems (following standard processing)
• Train new WSDC Ops Engineer
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
15
Backup Slides
Operational Readiness Review – October 7-8, 2009
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
16
Daily Processing Status
• Most Pessimistic Week – Max. load period; early ops
• Full data load • Scan/frame automation not ready yet • 5/40 coverage • Weekend free period inaccessible to operators • Full daily multi-scan pipelines and coadds generated
– Will probably not really be generated for a few weeks – Can run as few/many as we can afford
Operational Readiness Review – October 7-8, 2009
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
17
Daily Processing Status
Most Pessimistic Week; No Weekend Coverage
Operational Readiness Review – October 7-8, 2009
National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology
18
Daily Processing Status
• Most Pessimistic Week; No Weekend Coverage
• Transfers 98 hours Parallel 3.5 hr x 28/week • Ingest/QL 42 hours Parallel 1.5 hr x 28/week • Scan/frame 28 hours 4 hr x 7/week • Multi-scan 14 hours 2 hr x 7/week • Coadds (optional) 14 hours 2 hr x 7/week • WMOPS 8 hours 2 hr x 4/week • Backups/Maintenance 28 hours 4 hr x 7/week • Total 232 hours • Elapsed (clock time) 92 hours • Free/pad 76 hours • Non-weekend free/pad 28 hours (23%)
Operational Readiness Review – October 7-8, 2009
top related