Australian Virtual Observatory Pacific Rim Applications and Grid Middleware Assembly The 4th Workshop 5th-6th June 2003 Monash University David Barnes.

Australian Virtual Observatory

Pacific Rim Applications and Grid Middleware Assembly

The 4th Workshop 5th-6th June 2003 Monash University

David BarnesSchool of Physics, The University of Melbourne

What is a Virtual Observatory?

• A Virtual Observatory (VO) is a distributed, uniform interface to the data archives of the world’s major astronomical observatories.

• A VO is explored with advanced data mining and visualisation tools which exploit the unified interface to enable cross-correlation and combined processing of distributed and diverse datasets.

• VOs will rely on, and provide motivation for, the development of national and international computational and data grids.

Scientific motivation

• Understanding of astrophysical processes depends on multi-wavelength observations and input from theoretical models.

• As telescopes and instruments grow in complexity, surveys generate massive databases which require increasing expertise to comprehend.

• Theoretical modeling codes are growing in sophistication to consume available compute time.

• Major advances in astrophysics will be enabled by transparently cross-matching, cross-correlating and inter-processing otherwise disparate data.

Sample multi-wavelength data for the galaxy IC5332 (Ryan-Weber)

Visible blue light - young hot stars

Infrared light - old cooler stars

H-alpha spectral line - star forming sites

HI spectral line - gas to form stars

HI velocity field - kinematics

HI velocity dispersion - gas stability, parameters

Integrated HI spectrum - total neutral gas mass, distance from redshift

And this is just the data on one object from three

Australian telescopes!

Fundamental VO Challenges• Data description: multi-wavelength, multi-

resolution, multi-dimensional, multi-domain (optical, radio, X-ray, …), world coordinate systems, limited period ownership, …

• Data provision: distributed mass storage, high-bandwidth networks, registries, …

• Data processing: high performance clusters as grid nodes, data to code versus code to data, mountains of legacy software!, …

• Interface: portals, visual data flow control, analysis tools, display tools, …

Aus-VO structure 2003

• Phase A funded AUD 260K by a 2003 ARC grant:– The University of Melbourne– The University of Sydney– CSIRO Australia Telescope National Facility– Anglo-Australian Observatory

• Additional institutes participating w/o direct funding from the ARC grant:– ANU, Mount Stromlo Observatory & APAC

– CSIRO Mathematical and Information Sciences

– University of Queensland

– VPAC & GridBus (Melb)

• Lead investigator Rachel Webster (Melb)• Project scientist David Barnes (Melb)

Aus-VO projects 2003

• Common format on-line archive projects:– HIPASS catalog: HI Parkes All Sky Survey: neutral

Hydrogen spectral line survey, ~4,300 sources with 138 parameters and 1024-channel spectra

– SUMSS catalog: Sydney University Molonglo Sky Survey: radio continuum survey at 843 MHz, >100,000 sources

– 2dFGRS QSO catalog: 2-degree Field Galaxy Redshift Survey: optical spectra of >20,000 southern quasi-stellar objects

– ATCA archive: Australia Telescope Compact Array archive: all observations since 1988, circa 1.5 TB of more than 1,000 separate observing projects! Massive exercise in describing data with metadata.

– MACHO archive: Massive Compact Halo Objects archive: 8yr lightcurves for >18M stars

Aus-VO projects 2003• Server-based visualisation tools:

– client Java canvas for legacy software package AIPS++ to draw on from a remote server (ATNF)

– grid-service implementation of distributed volume rendering - remote data transferred to remote cluster, with display canvas applet supplied by coordinating portal

• Pipelines to enable on-line reprocessing of archived raw or pre-processed telescope data:– Molonglo Observatory Synthesis Telescope

• Interfaces: beta testers for the AstroGrid consortium software

VO Interface & Portal

• Agreement with AstroGrid (UK e-Science project) to be testers for their data publication and portal creation code.

• Collecting the necessary resources and intend to have an AstroGrid-based portal serving HIPASS catalog data for demonstration at IAU General Assembly in July 2003.

• Separately testing IBM Lotus Notes and Domino Server for publication of astronomical catalogs.

Grid-based Visualisation• ATNF will build a Java

PixelCanvas so that AIPS++ visualisation applications can be deployed as Web-Service and Grid- Service Java Applets

• AIPS++ is modern, OpenSource software for reducing (radio) astronomy data, 1.6M lines of code.

Grid-based Volume Rendering• Agreement between Melbourne and AstroGrid to develop our

existing distributed-data volume rendering code into a fully-fledged Grid-Service. [see my talk at GridBus this Saturday][see my talk at GridBus this Saturday]

• Challenge is to interactively render a multi-GB cube at the IAU GA 2003, using GridFTP to transfer the data volume from a remote data warehouse to a remote rendering cluster and display and control the rendering from an applet.

Time to render 512x512 view of 1024x1024x1024 volume (seconds)

1

10

100

1000

0 10 20 30 40

number of nodes

The near future: data grids for Aus-VOThe near future: data grids for Aus-VO

• Australian archives range from ~10 GB to ~10 TB in processed (reduced) size.

• providing just the processed images and spectra on-line requires a distributed, high-bandwidth network of data servers – that is, a data grid.

• users may want some simple operations such as smoothing or filtering, applied at the data server. This is a virtual data grid.

The near future: compute grids for Aus-VOThe near future: compute grids for Aus-VO

• More complex operations may be applied requiring significant processing:– source detection and parameterisation– reprocessing of raw or intermediate data

products with new calibration algorithms– combined processing of raw, intermediate or

"final product" data from different archives

• These operations require a distributed, high-bandwidth network of computational nodes – that is, a compute grid.

2004 ARC LIEF grant2004 ARC LIEF grant

• 10 partners!

• more data archives on-line

• more tools developed with special focus on server-based visualisation

• construction of the Australian Australian Astronomy Grid…Astronomy Grid…

The Australian Astronomy Grid 2004The Australian Astronomy Grid 2004

http://www.aus-vo.org