天文学におけるデータグリッド · 2019-12-06 · Exchange of Meta Data:OAI-PMH Searchable Registry Publishing Registry Publishing Registry Data Service Analysis Service

1

天文学におけるデータグリッド

Masatoshi Ohishi / NAOJ, Sokendai & NII大石雅寿 / 国立天文台，総合研究大学院大学，

国立情報学研究所

masatoshi.ohishi@nao.ac.jp

2

Supported by• JSPS

“Core to Core Program” (2004~2008)

• MEXT Grant-in-Aid“Information Explosion” (2001~ )

• National Institute for Informatics“CSI Program” (2007~ )

• NAOJ

March 19, 2008 グリッド協議会第23回ワークショップ

March 19, 2008 グリッド協議会第23回ワークショップ3

Data Resources in the World• ESO VLT @ Paranal• ESO telescopes @ La Silla• Telescopes in Mauna Kea

Subaru, Keck, JCMT, CSO, SMA,,,• Telescopes on board satellites

HST, GALEX,,,,

March 19, 2008 グリッド協議会第23回ワークショップ4

Data Resources in NAOJ• Subaru 8.2m Optical-Infrared Telescope• Kiso 105cm Schmidt Camera• Okayama 188cm Optical Telescope• Nobeyama 45m Radio Telescope• Nobeyama Millimeter Array• Nobeyama Radioheliograph• VSOP• VERA• ALMA

Subaru

Nobeyama 45m

March 19, 2008 グリッド協議会第23回ワークショップ5

Planned Data Resources• ALMA• JWST• LSST• LOFAR• SKA• Thirty MetreTelescope• Giant Magellan Telescope• European Extremely Large Telescope

TMT

ALMALSST

SKA

JWST

March 19, 2

グリッド協議会第23回ワークショップ6

Simple Spectrum Access

Image / Spectrum / Catalog interoperability

Crab nebula 7

Images obtained by Subaru 8

9

10

March 19, 2008

グリッド協議会第23回ワークショップ11

Data Productivity• Nobeyama Radio ：~1TB/yr• Subaru@Hawaii ：~20TB/yr• AKARI (IR satellite by JAXA)

：several 100 GB in total• ALMA(under construction) ：~PB/yr• LSST (US, planned)：~30 PB/yr

(~ 100 TB/night)

Flood of excellent data (survey data)Digitized & Archived Databases

Archival System for Subaru, located in Hilo, Hawaii ~ 120 TB

12

March 19, 2

グリッド協議会第23回ワークショップ13

Astronomy based on Archives

– large fraction of astro-papers based on archives– HST archive use growing faster than archive

graphics from US NVO project

Archives will be the core of the telescope operations systems and data processing systems

backup main streamsequential random

14

15

March 19, 2008 グリッド協議会第23回ワークショップ16

VO Projects in the world

• 15 countries and a region (EU)• International Virtual Observatory Alliance (IVOA)

Standards to interoperate VOs

• Meta data,data models,data accesses,output format,etc.

March 19, 2

グリッド協議会第23回ワークショップ17

March 19, 2008

グリッド協議会第23回ワークショップ18

IVOA Interoperability WS

• Twice a year• Discussions

toward standardization

• Human network as a basis for cyber network

Kyoto, 2005 May

March 19, 2008 グリッド協議会第23回ワークショップ 19

SkyNode1.0

SDSS QSO

… 2MASSPortal

P. Reg.

S. Reg.

LDAP

SkyNode 0.7.4

SIAP/SSAP

解析解析

解析AnalysisServers

MDS

P. Reg.

User StorageWeb Service

HTTP GetFTP

LDAPOther VOs

User Request

Invoke

Invoke

InvokeSe

rvic

e D

isco

very

Auth.

Metadata

Harvesting

JVO

Subaru

Grid System

Registry

Data Service

Overview of the JVO Portal Service

March 19, 2008 グリッド協議会第23回ワークショップ 20

Resource Metadata

• Resource Identification:– Title, ShortName, Identifier

• Curation:– Publisher, PublisherID, Creator, Creator.Logo,

Contributor, Date, Version, Contact.Name, Contact.Email

• General content:– Subject, Description, Source, ReferenceURL,

Type, ContentLevel, Relationship, RelationshipID

March 19, 2008 グリッド協議会第23回ワークショップ 21

Exchange of Meta Data:OAI-PMHSearchable Registry

Publishing Registry

Publishing Registry

Data Service

Analysis Service

Virtual Observatory Client

Query Services

Access to various services

March 19, 2008 グリッド協議会第23回ワークショップ 22

SkyNode1.0

SDSS QSO

… 2MASSPortal

P. Reg.

S. Reg.

LDAP

SkyNode 0.7.4

SIAP/SSAP

解析解析

解析AnalysisServers

MDS

P. Reg.

User StorageWeb Service

HTTP GetFTP

LDAPOther VOs

User Request

Invoke

Invoke

InvokeSe

rvic

e D

isco

very

Auth.

Metadata

Harvesting

JVO

Subaru

Grid System

Registry

Data Service

Overview of the JVO Portal Service

March 19, 2008 グリッド協議会第23回ワークショップ 23

Data Access Protocols• Parameter query in terms of the HTTP

Simple Image Access Protocol (SIAP)

Simple Spectrum Access Protocol (SSAP)

Simple Table Access Protocol (STAP)

etc.

•Unified query language (JVOQL) for both the catalog and observation data such as image data, spectrum, 3D-cube, photon list …

http://jvo.nao.ac.jp/imageData?Pos=24,5&Size=0.2&format=VOTable

Select imageURL, …From naoj:imageDataWhere pos=Point(24,5) and size=0.2 and format=‘VOTable’

March 19, 2008 グリッド協議会第23回ワークショップ 24

JVO SkyNode Architecture

JVOQL

JVOQLADQL

SIAPSSAP

HTTP/SOAPAXIS

HTTP/ParameterTomcat

GridGTK 3 (optional)

JDBC forSkyNode

SelectSQLTranslator

Query ExecuterFormatter

Internet JVO SkyNode

JVO Portal

• Apache Axis and Tomcat are used for implementing the VO standard interface, and Globus Toolkit is used for an optional Grid interface.

• Four kinds of query languages are converted to java class SelectSQL and query is executed through JDBC-like interface.

• Query results is obtained as ResultSet, and table data is formatted to VOTable or CSV file.

JVO SkyNode

DBMSResultSetgetMetadata()

getString()getFits()getJpeg()

March 19, 2008 グリッド協議会第23回ワークショップ 25

File Formats• Flexible Image Transfer System (FITS)

– standardized in early 80’s to exchange observed data

– 1 record = 2880 bytes– (Header, Data)(Header, Data)・・・

– IAU has the FITS WG to maintain its specification

• VOTable– used in Virtual Observatories as an output format– described in XML, and standardized in IVOA– can inline FITS files / contain a link to FITS files

User Portal26

http://jvo.nao.ac.jp/portal/By Shirasaki-san

March 19, 2008

グリッド協議会第23回ワークショップ

March 19, 2008 グリッド協議会第23回ワークショップ 27

MAG_B

Log(

F_IR

)

28

Spectrum Data Viewer• Display• Baseline

estimation• Line

detection & parameter read

• ・・・

March 19, 2008 グリッド協議会第23回ワークショップ

グリッド協議会第23回ワークショップ29

Interconnected VO Data services in the World

Canada VO

China VO

Data Service Operations have just been started this month.

March 19, 2008

グリッド協議会第23回ワークショップ30

What astronomers do …

• Query image data in optical and infrared

• Extract source parameters (position, size, elongation, brightness, etc.) from the images by using Sextractor

• Generate a personal catalog• Perform cross-matching to find

peculiar sources within the catalog

March 19, 2008 グリッド協議会第23回ワークショップ31

Improvement of Work Flow system

Items in FY2007• Improvement of WFDL

– Shorten the volume of DL– Array definition– Awk-like description (loop

description in a single line) – and so on

• Built-in functions– Available to register and

use built-in functions• Status and Log system• Need a GUI builder

– Investigating reference systems

WF Editor

Execution monitor

WF builder prototype by means of Taverna•Good graphical interface together with corresponding script

•Builder prototype in 2007•Easy construction of WFs w/ WFDL•Visual display of execution status

32

March 19, 2008グリッド協議会第23回ワークショップ

Towards the New Infrastructure for Astronomical Research

March 19, 2008

グリッド協議会第23回ワークショップ33

N E T W O R K

SMOKAALMA

East AsiaRegional Center

ALMA East AsiaRegional Center

NVO

CDS/VO-

France

AstroGrid

Euro-VO

Other VOs

Other datain Japan

JAXA/ISAS

Hinode

JAPANESE VIRTUAL OBSERVATORYJAPANESE VIRTUAL OBSERVATORY

WF

Data Analysis Engines・・・・

AKARI

Virtual Storage (VOSpace)

• 相互の計算資源を効率的に利用，CPUアーキテク

チャの違いを仮想化

• 共用のディスク領域

• 共通のアクセス方法 (WS / link / ,,,)

March 19, 2008

グリッド協議会第23回ワークショップ34

Virtual Storage with Common Access Interface

Japan VO US VO Euro VO UK VO

March 19, 2008

グリッド協議会第23回ワークショップ35

Dissemination of the VO Technology

• Data centers for astronomy in Japan (and East Asia)

– Visiting demonstrations– Regular training courses

• Related science fields– Planetary Science (c.f. IPDA)– Solar Terrestrial Environment– Informatics ~ DB researchers

ま と め

• 天文学では観測データを超大規模アーカイブ／データベースに格納し，研究に活用する

• 情報技術を天文学に応用することにより，世界中の天文データへのアクセスが極めて容易になった

データアクセス機能については，NAOJで超高速検索機能付き運用システムを2008年から運用開始した

• 収集したデータから，宇宙の新知見を見いだすために必要なデータ解析を容易に実行するためのWFシステ

ム構築が重要

36

March 19, 2

グリッド協議会第23回ワークショップ37

JVO collaborators

ScientistsNAOJ• Ohishi• Mizumoto• Oe• Shirasaki• Tanaka• (Honda)ICRR• YasudaAoyama Gakuin U.• Masunaga

System EngineersFujitsu Ltd.• Kawarai• Ishihara• Machida• Tsutsumi• Hiyama

SEC Ltd.• Morita• Nakamoto• Kobayashi• Sakamoto• Tsuyuki

SupporterNII•Miura