Future of Astronomy: enormous datasets, massive computing, innovative instrumentation Rachel Webster & David Barnes (Project Leader & Project Scientist,

Future of AstronomyFuture of Astronomy: : enormous datasets, enormous datasets, massive computing, massive computing,

innovative instrumentationinnovative instrumentation

Rachel Webster & David BarnesRachel Webster & David Barnes(Project Leader & Project Scientist, Australian (Project Leader & Project Scientist, Australian

Virtual Observatory)Virtual Observatory)

School of Physics, The University of School of Physics, The University of MelbourneMelbourne

TopicsTopics

1.1. Astronomy is a theoretical and observational Astronomy is a theoretical and observational science with science with massivemassive heterogeneous heterogeneous datasets.datasets.

2.2. The Virtual Observatory (VO)The Virtual Observatory (VO)

3.3. Aus-VO: the Australian projectAus-VO: the Australian project

4.4. A new local opportunity: MWA low frequency A new local opportunity: MWA low frequency arrayarray

Read here for a summary of each slide…..

2. What is a Virtual Observatory?2. What is a Virtual Observatory?

• A Virtual Observatory (VO) is a distributed, A Virtual Observatory (VO) is a distributed, uniform uniform interfaceinterface to the data archives of the to the data archives of the world’s world’s major astronomical facilitiesmajor astronomical facilities..

• A VO is realised with advanced data mining A VO is realised with advanced data mining and visualisation tools which exploit the unified and visualisation tools which exploit the unified interface to enable interface to enable cross-correlationcross-correlation and and combined processingcombined processing of distributed and of distributed and diverse datasets.diverse datasets.

• VOs will rely on, and provide motivation for, VOs will rely on, and provide motivation for, the development of the development of national and international national and international computational and data gridscomputational and data grids..

Virtual observatories will effect a “sea change” in the way astronomy is done.

International Data deluge!International Data deluge!• Dozens of new surveys 2003 to 2008Dozens of new surveys 2003 to 2008• Many (10 – 100) terabytes per surveyMany (10 – 100) terabytes per survey• 10 – 100 researchers per survey10 – 100 researchers per survey• International collaborations (almost always)International collaborations (almost always)• Data is non-proprietary (usually)Data is non-proprietary (usually)

Surveys are no longer within the scope of the solo researcher, and also cannot be accommodated by isolated computing and storage facilities.

Enter Grid Computing and the Virtual Enter Grid Computing and the Virtual ObservatoryObservatory

New surveys of the whole sky need a new paradigm: enter the Virtual Observatory.

3. Aus-VO and APAC Grid Project3. Aus-VO and APAC Grid Project

• 10 institutions; 4 large grants over 2-4 10 institutions; 4 large grants over 2-4 years (LIEF & APAC)years (LIEF & APAC)

• VO Data Warehousing (10 major VO Data Warehousing (10 major datasets)datasets)

• Gravity Wave Research GridGravity Wave Research Grid• VO Theory PortalVO Theory Portal• Registry, storage service, hpc, query Registry, storage service, hpc, query

languages, visualisation, data mininglanguages, visualisation, data mining• Melbourne-led (at present)Melbourne-led (at present)

The Australian Astronomy Grid will be developed to handle data storage and access needs.

IVOA and the International ContextIVOA and the International Context

• More than 15 active national VO More than 15 active national VO programs;programs;

• Multi-million $$ investments in UK, Multi-million $$ investments in UK, USA and EuropeUSA and Europe

• Loose but collegial collaboration Loose but collegial collaboration • Responsible for international Responsible for international

standardsstandards• Active meeting program (we have no Active meeting program (we have no

funds to participate)funds to participate)

The Australian Astronomy Grid will be developed to handle data storage and access needs.

Australian data storage and accessAustralian data storage and access

• Australian astronomy data holdings Australian astronomy data holdings presently exceed ~40 TB in size, and are presently exceed ~40 TB in size, and are growing rapidly.growing rapidly.

• Typical high-end workstations can store only Typical high-end workstations can store only ~100 GB or so.~100 GB or so.

• Providing access to the data – raw and Providing access to the data – raw and processed – requires a processed – requires a distributed, high-distributed, high-bandwidth network of data servers.bandwidth network of data servers.

• The Australian Virtual Observatory project is The Australian Virtual Observatory project is developing the developing the Australian Astronomy GridAustralian Astronomy Grid to to handle future demand.handle future demand.

The Australian Astronomy Grid will be developed to handle data storage and access needs.

The Australian Astronomy Grid 2004The Australian Astronomy Grid 2004

The HI Parkes All Sky SurveyThe HI Parkes All Sky Survey

• Parkes 64m radio Parkes 64m radio telescope in NSW.telescope in NSW.

• Hyperfine transition Hyperfine transition of atomic Hydrogen, of atomic Hydrogen, =21cm.=21cm.

• 280 days over 4 280 days over 4 years; 40 observers; years; 40 observers; 1000GB raw data1000GB raw data..

• 400 image “cubes” 400 image “cubes” searched by searched by computer for computer for significant signals.significant signals.

The Parkes telescope has surveyed the entire southern sky for emission from Hydrogen.

The Two Micron All Sky SurveyThe Two Micron All Sky Survey

• 4M images4M images• 470M point sources470M point sources• 1.6M extended sources1.6M extended sources• ~500 parameters per source!~500 parameters per source!• 25 TB of data!25 TB of data!

All-sky map of 1.6 million 2MASS extended sources.

Another example is 2MASS which has catalogued nearly half a billion objects in the sky.

Jarrett et al., 2000

Less than 10% of the catalogue fits in memory on a typical workstation

Other major surveys...Other major surveys...

• Sloan Digital Sky Survey Sloan Digital Sky Survey (SDSS)(SDSS)– position and brightness of 100M objectsposition and brightness of 100M objects– distance to more than 100K quasarsdistance to more than 100K quasars– 15 Terabytes of data!15 Terabytes of data!

• Radial Velocity Experiment Radial Velocity Experiment (RAVE)(RAVE)– 50M stars: velocities, metallicities, and abundance 50M stars: velocities, metallicities, and abundance

ratiosratios– 10 TB of data!10 TB of data!

• Faint Images of the Radio Sky Faint Images of the Radio Sky (FIRST)(FIRST)– 811,000 sources with radio continuum flux densities 811,000 sources with radio continuum flux densities

at 20cm wavelengthat 20cm wavelength

Dozens of major, terabyte-scale survey projects are underway or planned.

Theoretical AstronomyTheoretical Astronomy

• Theory provides models of Theory provides models of the phenomena discovered the phenomena discovered by observations.by observations.

• Theory makes predictions of Theory makes predictions of what will be seen by future what will be seen by future facilities.facilities.

• Many theories are non-Many theories are non-analytic, and sophisticated analytic, and sophisticated numerical simulations are run numerical simulations are run on supercomputers to on supercomputers to produce realisations of produce realisations of synthetic universes.synthetic universes.

Simulations can produce realisations of synthetic universes from fundamental physics.

Linking theory to observationsLinking theory to observations

• Simulations are not expected to produce our Simulations are not expected to produce our particular Universe.particular Universe.

• Instead, they generate systems which can be Instead, they generate systems which can be compared compared statisticallystatistically to our Universe. to our Universe.

• Realisations of a good model should be Realisations of a good model should be statistically indistinguishable from the statistically indistinguishable from the observed Universe.observed Universe.

• Useful statistical comparisons demand high Useful statistical comparisons demand high quality data and large numbers of objects quality data and large numbers of objects independent of how you bin the data.independent of how you bin the data.

• Deeper, faster and more sophisticated surveys Deeper, faster and more sophisticated surveys are called for...are called for...

Bigger and better simulations demand super surveys for statistical comprehension.

4. MWA: Mileura Widefield Array4. MWA: Mileura Widefield Array

• Low frequency radio domain (110-240 Low frequency radio domain (110-240 MHz) largely unexploredMHz) largely unexplored

• Not easy: ionosphere, FM band, etcNot easy: ionosphere, FM band, etc• BUT: aim to detect the first sources BUT: aim to detect the first sources

and map Epoch of Reionisationand map Epoch of Reionisation• One of 3 international experimentsOne of 3 international experiments

(strongest project (strongest project ))

New US/Australian low frequency array

• Remote WA, for first light in 2007Remote WA, for first light in 2007• 6TB fibre link to Geraldton6TB fibre link to Geraldton• Storage: 100’s TBsStorage: 100’s TBs• CPU: 50TflopsCPU: 50Tflops

• Melbourne, in collaboration with MIT, Melbourne, in collaboration with MIT, ATNF, Harvard and othersATNF, Harvard and others

• Industry partnersIndustry partners

New low frequency array will use innovative data-handling algorithms

MWA: Basic ApproachMWA: Basic Approach

‘Desert Australia’ is probably the best site in the world for low frequency astronomy

MWA: Signal ProcessingMWA: Signal Processing

• 500 tiles (x16 dipoles)500 tiles (x16 dipoles)• 125,000 baselines, 4 polarization products125,000 baselines, 4 polarization products• FPGA based hardwareFPGA based hardware• Receiver: analog and mixed-signal front Receiver: analog and mixed-signal front

end; digital back end end; digital back end • Data stream: ~2 billion visibilities/0.5 secData stream: ~2 billion visibilities/0.5 sec

Technical requirements and directions

Melbourne Astrophysics RequirementsMelbourne Astrophysics Requirements

• High Bandwidth Communications High Bandwidth Communications (Access Grid): scientific collaboration & (Access Grid): scientific collaboration & conferencesconferences

• Functional Grid: storage & processingFunctional Grid: storage & processing

• Institutional Commitment: planning, Institutional Commitment: planning, resourcing, r&d,resourcing, r&d,

• Institutional Leadership: NEWInstitutional Leadership: NEW