1 天文学におけるデータグリッド Masatoshi Ohishi / NAOJ, Sokendai & NII 大石雅寿 / 国立天文台,総合研究大学院大学, 国立情報学研究所 [email protected]
1
天文学におけるデータグリッド
Masatoshi Ohishi / NAOJ, Sokendai & NII大石雅寿 / 国立天文台,総合研究大学院大学,
国立情報学研究所
2
Supported by• JSPS
“Core to Core Program” (2004~2008)
• MEXT Grant-in-Aid“Information Explosion” (2001~ )
• National Institute for Informatics“CSI Program” (2007~ )
• NAOJ
March 19, 2008 グリッド協議会第23回ワークショップ
March 19, 2008 グリッド協議会第23回ワークショップ3
Data Resources in the World• ESO VLT @ Paranal• ESO telescopes @ La Silla• Telescopes in Mauna Kea
Subaru, Keck, JCMT, CSO, SMA,,,• Telescopes on board satellites
HST, GALEX,,,,
March 19, 2008 グリッド協議会第23回ワークショップ4
Data Resources in NAOJ• Subaru 8.2m Optical-Infrared Telescope• Kiso 105cm Schmidt Camera• Okayama 188cm Optical Telescope• Nobeyama 45m Radio Telescope• Nobeyama Millimeter Array• Nobeyama Radioheliograph• VSOP• VERA• ALMA
Subaru
Nobeyama 45m
March 19, 2008 グリッド協議会第23回ワークショップ5
Planned Data Resources• ALMA• JWST• LSST• LOFAR• SKA• Thirty MetreTelescope• Giant Magellan Telescope• European Extremely Large Telescope
TMT
ALMALSST
SKA
JWST
March 19, 2
グリッド協議会第23回ワークショップ6
Simple Spectrum Access
Image / Spectrum / Catalog interoperability
Crab nebula 7
Images obtained by Subaru 8
9
10
March 19, 2008
グリッド協議会第23回ワークショップ11
Data Productivity• Nobeyama Radio :~1TB/yr• Subaru@Hawaii :~20TB/yr• AKARI (IR satellite by JAXA)
:several 100 GB in total• ALMA(under construction) :~PB/yr• LSST (US, planned):~30 PB/yr
(~ 100 TB/night)
Flood of excellent data (survey data)Digitized & Archived Databases
Archival System for Subaru, located in Hilo, Hawaii ~ 120 TB
12
March 19, 2
グリッド協議会第23回ワークショップ13
Astronomy based on Archives
– large fraction of astro-papers based on archives– HST archive use growing faster than archive
graphics from US NVO project
Archives will be the core of the telescope operations systems and data processing systems
backup main streamsequential random
14
15
March 19, 2008 グリッド協議会第23回ワークショップ16
VO Projects in the world
• 15 countries and a region (EU)• International Virtual Observatory Alliance (IVOA)
Standards to interoperate VOs
• Meta data,data models,data accesses,output format,etc.
March 19, 2
グリッド協議会第23回ワークショップ17
March 19, 2008
グリッド協議会第23回ワークショップ18
IVOA Interoperability WS
• Twice a year• Discussions
toward standardization
• Human network as a basis for cyber network
Kyoto, 2005 May
March 19, 2008 グリッド協議会第23回ワークショップ 19
SkyNode1.0
SDSS QSO
… 2MASSPortal
P. Reg.
S. Reg.
LDAP
SkyNode 0.7.4
SIAP/SSAP
解析解析
解析AnalysisServers
MDS
P. Reg.
User StorageWeb Service
HTTP GetFTP
LDAPOther VOs
User Request
Invoke
Invoke
InvokeSe
rvic
e D
isco
very
Auth.
Metadata
Harvesting
JVO
Subaru
Grid System
Registry
Data Service
Overview of the JVO Portal Service
March 19, 2008 グリッド協議会第23回ワークショップ 20
Resource Metadata
• Resource Identification:– Title, ShortName, Identifier
• Curation:– Publisher, PublisherID, Creator, Creator.Logo,
Contributor, Date, Version, Contact.Name, Contact.Email
• General content:– Subject, Description, Source, ReferenceURL,
Type, ContentLevel, Relationship, RelationshipID
March 19, 2008 グリッド協議会第23回ワークショップ 21
Exchange of Meta Data:OAI-PMHSearchable Registry
Publishing Registry
Publishing Registry
Data Service
Analysis Service
Virtual Observatory Client
Query Services
Access to various services
March 19, 2008 グリッド協議会第23回ワークショップ 22
SkyNode1.0
SDSS QSO
… 2MASSPortal
P. Reg.
S. Reg.
LDAP
SkyNode 0.7.4
SIAP/SSAP
解析解析
解析AnalysisServers
MDS
P. Reg.
User StorageWeb Service
HTTP GetFTP
LDAPOther VOs
User Request
Invoke
Invoke
InvokeSe
rvic
e D
isco
very
Auth.
Metadata
Harvesting
JVO
Subaru
Grid System
Registry
Data Service
Overview of the JVO Portal Service
March 19, 2008 グリッド協議会第23回ワークショップ 23
Data Access Protocols• Parameter query in terms of the HTTP
Simple Image Access Protocol (SIAP)
Simple Spectrum Access Protocol (SSAP)
Simple Table Access Protocol (STAP)
etc.
•Unified query language (JVOQL) for both the catalog and observation data such as image data, spectrum, 3D-cube, photon list …
http://jvo.nao.ac.jp/imageData?Pos=24,5&Size=0.2&format=VOTable
Select imageURL, …From naoj:imageDataWhere pos=Point(24,5) and size=0.2 and format=‘VOTable’
March 19, 2008 グリッド協議会第23回ワークショップ 24
JVO SkyNode Architecture
JVOQL
JVOQLADQL
SIAPSSAP
HTTP/SOAPAXIS
HTTP/ParameterTomcat
GridGTK 3 (optional)
JDBC forSkyNode
SelectSQLTranslator
Query ExecuterFormatter
Internet JVO SkyNode
JVO Portal
• Apache Axis and Tomcat are used for implementing the VO standard interface, and Globus Toolkit is used for an optional Grid interface.
• Four kinds of query languages are converted to java class SelectSQL and query is executed through JDBC-like interface.
• Query results is obtained as ResultSet, and table data is formatted to VOTable or CSV file.
JVO SkyNode
DBMSResultSetgetMetadata()
getString()getFits()getJpeg()
March 19, 2008 グリッド協議会第23回ワークショップ 25
File Formats• Flexible Image Transfer System (FITS)
– standardized in early 80’s to exchange observed data
– 1 record = 2880 bytes– (Header, Data)(Header, Data)・・・
– IAU has the FITS WG to maintain its specification
• VOTable– used in Virtual Observatories as an output format– described in XML, and standardized in IVOA– can inline FITS files / contain a link to FITS files
User Portal26
http://jvo.nao.ac.jp/portal/By Shirasaki-san
March 19, 2008
グリッド協議会第23回ワークショップ
March 19, 2008 グリッド協議会第23回ワークショップ 27
MAG_B
Log(
F_IR
)
28
Spectrum Data Viewer• Display• Baseline
estimation• Line
detection & parameter read
• ・・・
March 19, 2008 グリッド協議会第23回ワークショップ
グリッド協議会第23回ワークショップ29
Interconnected VO Data services in the World
Canada VO
China VO
Data Service Operations have just been started this month.
March 19, 2008
グリッド協議会第23回ワークショップ30
What astronomers do …
• Query image data in optical and infrared
• Extract source parameters (position, size, elongation, brightness, etc.) from the images by using Sextractor
• Generate a personal catalog• Perform cross-matching to find
peculiar sources within the catalog
March 19, 2008 グリッド協議会第23回ワークショップ31
Improvement of Work Flow system
Items in FY2007• Improvement of WFDL
– Shorten the volume of DL– Array definition– Awk-like description (loop
description in a single line) – and so on
• Built-in functions– Available to register and
use built-in functions• Status and Log system• Need a GUI builder
– Investigating reference systems
WF Editor
Execution monitor
WF builder prototype by means of Taverna•Good graphical interface together with corresponding script
•Builder prototype in 2007•Easy construction of WFs w/ WFDL•Visual display of execution status
32
March 19, 2008グリッド協議会第23回ワークショップ
Towards the New Infrastructure for Astronomical Research
March 19, 2008
グリッド協議会第23回ワークショップ33
N E T W O R K
SMOKAALMA
East AsiaRegional Center
ALMA East AsiaRegional Center
NVO
CDS/VO-
France
AstroGrid
Euro-VO
Other VOs
Other datain Japan
JAXA/ISAS
Hinode
JAPANESE VIRTUAL OBSERVATORYJAPANESE VIRTUAL OBSERVATORY
WF
Data Analysis Engines・・・・
AKARI
Virtual Storage (VOSpace)
• 相互の計算資源を効率的に利用,CPUアーキテク
チャの違いを仮想化
• 共用のディスク領域
• 共通のアクセス方法 (WS / link / ,,,)
March 19, 2008
グリッド協議会第23回ワークショップ34
Virtual Storage with Common Access Interface
Japan VO US VO Euro VO UK VO
March 19, 2008
グリッド協議会第23回ワークショップ35
Dissemination of the VO Technology
• Data centers for astronomy in Japan (and East Asia)
– Visiting demonstrations– Regular training courses
• Related science fields– Planetary Science (c.f. IPDA)– Solar Terrestrial Environment– Informatics ~ DB researchers
ま と め
• 天文学では観測データを超大規模アーカイブ/データベースに格納し,研究に活用する
• 情報技術を天文学に応用することにより,世界中の天文データへのアクセスが極めて容易になった
データアクセス機能については,NAOJで超高速検索機能付き運用システムを2008年から運用開始した
• 収集したデータから,宇宙の新知見を見いだすために必要なデータ解析を容易に実行するためのWFシステ
ム構築が重要
36
March 19, 2
グリッド協議会第23回ワークショップ37
JVO collaborators
ScientistsNAOJ• Ohishi• Mizumoto• Oe• Shirasaki• Tanaka• (Honda)ICRR• YasudaAoyama Gakuin U.• Masunaga
System EngineersFujitsu Ltd.• Kawarai• Ishihara• Machida• Tsutsumi• Hiyama
SEC Ltd.• Morita• Nakamoto• Kobayashi• Sakamoto• Tsuyuki
SupporterNII•Miura