DMD Data Flow Operations David Silva Data Management and Operations Division

ESO-Wide Review 2005

DMD Data Flow Operations

David SilvaData Management and Operations Division

ESO-Wide Review 2005

http://archive.eso.orghttp://www.eso.org/qc

Mission: ESO Archive Operations

• Data ingestion & delivery– Chile sites, WFCAM, surveys

• Data quality control and processing (Paranal)

• Archive/database content management– Data Interface Control Board

(DICB) chairman, members

• Archive systems administration• Database administration• HST and VO support• Remedy and Web administration

ESO-Wide Review 2005

DFO Personnel

Missing: Mignani (QC),Percheron (QC), Sjøberg

14 ISM, 11 CFTE

ESO-Wide Review 2005

2004 Highlights

• Department re-organization– Good-bye and thanks to Benoît

Pirenne!– Merged OTS and DFO/QC– Absorbed Web/Remedy

administration– Transition to Service Contract

• Before: many 1-on-1 contracts• Now: Data Conversion Services • Duration: 3/2004 - 3/2007 (option:

3/ 2009)• One year summary: going well

• New services– ESO library database interface– Proposal abstracts on-line– Archive query forms for [FILL IN]– Data Interface Control Document 3.0– QC services for MIDI

• Selected numbers– More than 100 hardware servers– Over 8 hours of Venus Transit Web

event, 1.5 TB and 45 million web hits served

– Ops_log database warehouse reached 1 billion rows in March 2004

– Archived ~1 million files, approximately 1/3 of all files in Archive

– Over 150 000 QC processing jobs (VIMOS = 54 000), 5 Tbytes processed, 1364 DVDs delivered

ESO-Wide Review 2005

2005 Outlook

• Archive opening to world– Per endorsement by Council– Monday 4 April 2005– Improved Archive interface(s)

(see VOS presentation)

• WFCAM data ingestion– UK in-kind, raw data, 6 TB/year

• New systems– Database server migration

• New hardware, Sybase ASE 12.5.3

– Web servers to Linux– New compute cluster– New DFO/QC tools– Net-based data transfer (again?)

• New QC Services– Apr: SINFONI, VISIR– Oct: AMBER– Improved science products– Expand to Visitor Mode (?)

• New Archive services– Support for VO

• Collaborate with VOS department

– Better ToO/RRM support– Better SCI/CAL association– Ingest & serve science products

• Prepare for VST/OmegaCam– Date volume/processing

accelerando begins…

ESO-Wide Review 2005

Data Volume Challenge

0.00

200.00

400.00

600.00

800.00

1000.00

1200.00

1400.00

1600.00

1800.00

Tera

Byt

es (

com

pre

ssed

)

2004 2005 2006 2007 2008 2009 2010 2011 2012

Other WFCAM VST VISTA ALMA

HST: 0.2 TB/yearVSTCam: 30 TB/year VISTA: 150 TB/year ALMA: 180 TB/year

ESO-Wide Review 2005

Data Processing Challenge

• Drivers: volume, complexity, demand– More pixels = more FLOPS

• Example: Cam, VISTA

– More processing per pixel = more FLOPS• Example: VIMOS, SINFONI

– More VO re-processing = more FLOPS

• Can be: volume and complexity– In terms of required processing

power: SINFONI:MUSE = 24

FLOPS = floating-point operations per second

• Required DFO processing power – UVES: 1.0 (actual, inc. fibre)– VIMOS: 2.2 (actual)– All VLT/VLTI (2004): ~ 6– Cam: ~ 21– VISTA: ~ 49– Based on Wicenec, Hanuschik,

Knudstrup, & Dolensky, 2005)

• Related issues– Peak loads– VO demand– 2nd generation VLT/VLTI instruments

ESO-Wide Review 2005

ESO Data Center: 2010

Storage volume: 1.2 PetaByte (one copy)Inflow: ~0.25 PB/yearMix of slow-access and fast-access mediaNeed 2nd secure copy

In addition: database server(s), etc.

100+ processors: 2006: 100 UVES equivalent 2010: 1000 UVES equivalent

Floor space: ~100 m2