Issue/Revision: 1.0 Reference: Gaia Archive – VO inside Status: Issued ESA UNCLASSIFIED - Releasable to the Public Gaia Archive operations for DR1 J. González-Núñez, J. Salgado, R. Gutiérrez-Sánchez, J.C. Segovia, J. Durán, C. Arviset, R. Alvarez, R. Gil, E. Anglada, J. Bakkers, U. Lammers ESA Science Data Centre 23/10/2016
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Issue/Revision: 1.0
Reference: Gaia Archive – VO inside
Status: Issued
ESA UNCLASSIFIED - Releasable to the Public
Gaia Archive operations for DR1
J. González-Núñez, J. Salgado, R. Gutiérrez-Sánchez, J.C. Segovia, J. Durán, C. Arviset, R. Alvarez, R. Gil, E. Anglada, J. Bakkers, U. Lammers ESA Science Data Centre 23/10/2016
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 2
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
14th September 2016
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 3
ESA UNCLASSIFIED - Releasable to the Public
The AS Role
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 4
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
High level scalability measures Ø Replication of data with Associate & Affiliate data centres
Ø AIP, ARI, ASDC, CDS, IRSA, GAVO
Ø Bulk download infrastructure based on a Content Delivery Network
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 5
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 6
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
RDBMS
Distributed Data Repository
science archive
TAP+
Browser GUI
TAP+, VOSpace
Programmatic
VO Apps
TAP+
VO s
ervi
ces
SAMP
Euro-VO Registry
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 7
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Scaling up S*AP vs scaling up TAP Ø Potentially large processing time for requests Ø Requests can be quite CPU intensive, both front-end and DB sides
Scaling up TAP+ Ø Persistent uploads Ø Server-side crossmatches
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 8
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Cloud vs dedicated HW Ø Dedicated HW can be selected and configured in ways Cloud of virtualized
infrastructures can’t Ø But it sets limits to scablability
Infrastructure dimensioning Ø Conservative engineering considering worst case scenarios Ø High performance hardware Ø Measures to ensure graceful service degradation if capacity is exceeded
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 9
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Stress test campaign Ø Stress test plan defined covering all server-side elements of the Gaia
Archive developed at ESAC (GACS), ie. TAP+ interface Ø Built with support of IT support team (SITU) & vendors engineering
teams support. Ø Executed in 4 iterations (IT1, IT2, IT3, IT4) with an increasing level of
systems monitoring Ø Full system performance analysis at each Iti Ø Incremental system infrastructure implemented to IT(i+1)
Ø Tested high data volume scenarios, with good performance up to network limits Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation factor)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 10
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Resource limits
Sync rows
Async rows
Sync time
Async time
DB space
Job results
Anonymous 100K 100K 1min 30min none N/A (np)
Registered unlimited unlimited 1min* 30min* 1GB* 1GB*
(np) not persistent * Can be upgraded on demand
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 11
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Job Scheduling
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 12
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 13
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Archive UI usage: First 24 hours
• Usage sessions: 12,005 • # of users: 10,959
UI Usage
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 14
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
File CDN download: First 15 days
• Total volume downloaded: 73 TB • # download requests: 9 millions
File download
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 15
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
• Number of queries • Synchronous: 174,807 • Asynchronous: 95,090
TAP Interface
TAP Interface: First 15 days
Related Documents
